Robotic multi-gripper assemblies and methods for gripping and holding objects

ABSTRACT

A system and method for operating a transport robot to simultaneously grasp and transfer multiple objects is disclosed. The transport robot includes a multi-gripper assembly having an array of addressable vacuum regions each configured to independently provide a vacuum. The robotic system receives image data representative of a group of objects. Individual target objects are identified in the group based on the received image data. Addressable vacuum regions are selected based on the identified target objects. The transport robot is command to cause the selected addressable vacuum regions to simultaneously grasp and transfer multiple target objects.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication Ser. No. 62/889,562, filed Aug. 21, 2019, which isincorporated herein by reference in its entirety.

This application contains subject matter related to U.S. patentapplication Ser. No. 16/855,751, filed Apr. 22, 2020, titled “ROBOTICMULTI-GRIPPER ASSEMBLIES AND METHODS FOR GRIPPING AND HOLDING OBJECTS,”which is incorporated herein by reference in its entirety.

TECHNICAL FIELD

The present technology is directed generally to robotic systems and,more specifically, robotic multi-grippers assemblies configured toselectively grip and hold objects.

BACKGROUND

Robots (e.g., machines configured to automatically/autonomously executephysical actions) are now extensively used in many fields. Robots, forexample, can be used to execute various tasks (e.g., manipulate ortransfer an object) in manufacturing, packaging, transport and/orshipping, etc. In executing the tasks, robots can replicate humanactions, thereby replacing or reducing human involvements that areotherwise required to perform dangerous or repetitive tasks. Robotsoften lack the sophistication necessary to duplicate human sensitivityand/or adaptability required for executing more complex tasks. Forexample, robots often have difficulty selectively gripping object(s)from a group of objects with immediately neighboring objects, as well asirregular shaped/sized objects, etc. Accordingly, there remains a needfor improved robotic systems and techniques for controlling and managingvarious aspects of the robots.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example environment in which a robotic systemtransports objects in accordance with one or more embodiments of thepresent technology.

FIG. 2 is a block diagram illustrating the robotic system in accordancewith one or more embodiments of the present technology.

FIG. 3 illustrates a multi-component transfer assembly in accordancewith one or more embodiments of the present technology.

FIG. 4 is a front view of an end effector coupled to a robotic arm of atransport robot in accordance with one or more embodiments of thepresent technology.

FIG. 5 is a bottom view of the end effector of FIG. 4.

FIG. 6 is a functional block diagram of a robotic transfer assembly inaccordance with one or more embodiments of the present technology.

FIG. 7 is a front, top isometric view of an end effector with amulti-gripper assembly in accordance with one or more embodiments of thepresent technology.

FIG. 8 is a front, bottom isometric view of the end effector of FIG. 7.

FIG. 9 is an exploded front isometric view of components of a vacuumgripper assembly with one or more embodiments of the present technology.

FIG. 10 is an isometric view of an assembly of vacuum grippers inaccordance with one or more embodiments of the present technology.

FIG. 11 is a top plan view of the assembly of FIG. 10.

FIG. 12 is an isometric view of an assembly of vacuum grippers inaccordance with one or more embodiments of the present technology.

FIG. 13 is an isometric view of a multi-gripper assembly in accordancewith another embodiment of the present technology.

FIG. 14 is an exploded isometric view of the multi-gripper assembly ofFIG. 13.

FIG. 15 is a partial cross-sectional view of a portion of amulti-gripper assembly in accordance with one or more embodiments of thepresent technology.

FIG. 16 is a flow diagram for operating a robotic system in accordancewith some embodiments of the present technology.

FIG. 17 is another flow diagram for operating a robotic system inaccordance with one or more embodiments of the present technology.

FIGS. 18-21 illustrate stages of robotically gripping and transportingobjects in accordance with one or more embodiments of the presenttechnology.

FIG. 22 illustrates example aspects of a grasp set in accordance withone or more embodiments of the present technology.

FIGS. 23A-23F illustrate example scenarios for simultaneouslytransferring multiple objects in accordance with one or more embodimentsof the present technology.

FIG. 24 illustrates example gripper placement conditions in accordancewith one or more embodiments of the present technology.

FIG. 25 illustrates an example task location in accordance with one ormore embodiments of the present technology.

FIG. 26 is another flow diagram for operating a robotic system inaccordance with one or more embodiments of the present technology.

DETAILED DESCRIPTION

Systems and methods for gripping selected objects are described herein.The systems can include a transport robot with multi-gripper assembliesconfigured to be operated independently or in conjunction togrip/release a single object or a plurality of objects. For example, thesystems can pick up multiple objects at the same time or sequentially.The system can select objects to be carried based upon, for example, thecarrying capability of the multi-gripper assembly, a transport plan, orcombinations thereof. The multi-gripper assembly can reliably gripobjects from a group of objects, irregular objects, shaped/sizedobjects, etc. For example, the multi-gripper assemblies can includeaddressable vacuum regions or banks each configured to draw in air suchthat only selected objects are held via a vacuum grip. The multi-gripperassembly can be robotically moved to transport the retained objects to adesired location and can then release the objects. The system can alsorelease gripped objects at the same time or sequentially. This processcan be repeated to transport any number of objects between differentlocations.

At least some embodiments are directed to a method for operating atransport robot having a multi-gripper assembly with addressable pick-upregions. The pick-up regions can be configured to independently providevacuum gripping. Target object(s) are identified based on captured imagedata. The pick-up regions can draw in air to grip the identified targetobject(s). In some embodiments, a transport robot can robotically movethe multi-gripper assembly, which is carrying the identified targetobjects.

In some embodiments, a robotic transport system includes a roboticapparatus, a target object detector, and a vacuum gripper device. Thevacuum gripper device includes a plurality of addressable regions and amanifold assembly. The manifold assembly can be fluidically coupled toeach of the addressable regions and to at least one vacuum line suchthat each addressable region is capable of independently providing anegative pressure via an array of suction elements. The negativepressure can be sufficient to hold at least one target object againstthe vacuum gripper device while the robotic apparatus moves the vacuumgripper device between different locations.

A method for operating a transport robot includes receiving image datarepresentative of a group of objects (e.g., a stack or pile of objects).One or more target objects are identified in the group based on thereceived image data. Addressable vacuum regions are selected based onthe identified one or more target objects. The transport robot iscommand to cause the selected vacuum regions to hold and transport theidentified one or more target objects. The transport robot includes amulti-gripper assembly having an array of vacuum regions each configuredto independently provide vacuum gripping. A vision sensor device cancapture the image data, which is representative of the target objectsadjacent to or held by the vacuum gripper device

In the following, numerous specific details are set forth to provide athorough understanding of the presently disclosed technology. In otherembodiments, the techniques introduced here can be practiced withoutthese specific details. In other instances, well-known features, such asspecific functions or routines, are not described in detail in order toavoid unnecessarily obscuring the present disclosure. References in thisdescription to “an embodiment,” “one embodiment,” or the like mean thata particular feature, structure, material, or characteristic beingdescribed is included in at least one embodiment of the presentdisclosure. Thus, the appearances of such phrases in this specificationdo not necessarily all refer to the same embodiment. On the other hand,such references are not necessarily mutually exclusive either.Furthermore, the particular features, structures, materials, orcharacteristics can be combined in any suitable manner in one or moreembodiments. It is to be understood that the various embodiments shownin the figures are merely illustrative representations and are notnecessarily drawn to scale.

Several details describing structures or processes that are well-knownand often associated with robotic systems and subsystems, but that canunnecessarily obscure some significant aspects of the disclosedtechniques, are not set forth in the following description for purposesof clarity. Moreover, although the following disclosure sets forthseveral embodiments of different aspects of the present technology,several other embodiments can have different configurations or differentcomponents than those described in this section. Accordingly, thedisclosed techniques can have other embodiments with additional elementsor without several of the elements described below.

Many embodiments or aspects of the present disclosure described belowcan take the form of computer- or controller-executable instructions,including routines executed by a programmable computer or controller.Those skilled in the relevant art will appreciate that the disclosedtechniques can be practiced on computer or controller systems other thanthose shown and described below. The techniques described herein can beembodied in a special-purpose computer or data processor that isspecifically programmed, configured, or constructed to execute one ormore of the computer-executable instructions described below.Accordingly, the terms “computer” and “controller” as generally usedherein refer to any data processor and can include Internet appliancesand handheld devices (including palm-top computers, wearable computers,cellular or mobile phones, multi-processor systems, processor-based orprogrammable consumer electronics, network computers, mini computers,and the like). Information handled by these computers and controllerscan be presented at any suitable display medium, including a liquidcrystal display (LCD). Instructions for executing computer- orcontroller-executable tasks can be stored in or on any suitablecomputer-readable medium, including hardware, firmware, or a combinationof hardware and firmware. Instructions can be contained in any suitablememory device, including, for example, a flash drive, USB device, and/orother suitable medium, including a tangible, non-transientcomputer-readable medium.

The terms “coupled” and “connected,” along with their derivatives, canbe used herein to describe structural relationships between components.It should be understood that these terms are not intended as synonymsfor each other. Rather, in particular embodiments, “connected” can beused to indicate that two or more elements are in direct contact witheach other. Unless otherwise made apparent in the context, the term“coupled” can be used to indicate that two or more elements are ineither direct or indirect (with other intervening elements between them)contact with each other, or that the two or more elements co-operate orinteract with each other (e.g., as in a cause-and-effect relationship,such as for signal transmission/reception or for function calls), orboth.

Suitable Environments

FIG. 1 is an illustration of an example environment in which a roboticsystem 100 transports objects. The robotic system 100 can include anunloading unit 102, a transfer unit or assembly 104 (“transfer assembly104”), a transport unit 106, a loading unit 108, or a combinationthereof in a warehouse or a distribution/shipping hub. Each of the unitsof the robotic system 100 can be configured to execute one or moretasks. The tasks can be combined in sequence to perform an operationthat achieves a goal, such as to unload objects from a truck or a vanfor storage in a warehouse or to unload objects from storage locationsand load them onto a truck or a van for shipping. In another example,the task can include moving objects from one container to anothercontainer. Each of the units can be configured to execute a sequence ofactions (e.g., operating one or more components therein) to execute atask.

In some embodiments, the task can include manipulation (e.g., movingand/or reorienting) of a target object or package 112 (e.g., boxes,cases, cages, pallets, etc.) from a start location 114 to a tasklocation 116. For example, the unloading unit 102 (e.g., a devanningrobot) can be configured to transfer the target package 112 from alocation in a carrier (e.g., a truck) to a location on a conveyor belt.The transfer assembly 104 (e.g., a palletizing robot assembly) can beconfigured to load packages 112 onto the transport unit 106 or conveyor120. In another example, the transfer assembly 104 can be configured totransfer one or more target packages 112 from one container to anothercontainer. The transfer assembly 104 can include a robotic end effector140 (“end effector 140”) with vacuum grippers (or vacuum regions) eachindividually operated to pick up and carry object(s) 112. When the endeffector 140 is placed adjacent an object, air can be into thegripper(s) adjacent to target packages 112, thereby creating a pressuredifferential sufficient for retaining the target objects. The targetobjects can be picked up and transported without damaging or marring theobject surfaces. The number of packages 112 carried at one time can beselected based upon stacking arrangements of objects at the pick-uplocation, available space at the drop off location, transport pathsbetween pick-up and drop off locations, optimization routines (e.g.,routines for optimizing unit usage, robotic usage, etc.), combinationsthereof, or the like. The end effector 140 can have one or more sensorsconfigured to output readings indicating information about retainedobjects (e.g., number and configurations of retained objects), relativepositions between any retained objects, or the like.

An imaging system 160 can provide image data used to monitor operationof components, identify target objects, track objects, or otherwiseperform tasks. The image data can be analyzed to evaluate, for example,package stacking arrangements (e.g., stacked packages, such as carboardboxes, packing containers, etc.), positional information of objects,available transport paths (e.g., transport paths between pickup zonesand drop off zones), positional information about gripping assemblies,or combinations thereof. A controller 109 can communicate with theimaging system 160 and other components of the robotic system 100. Thecontroller 109 can generate transport plans that include a sequence forpicking up and dropping off objects (e.g., illustrated as stablecontainers), positioning information, order information for picking upobjects, order information for dropping off objects, stacking plans(e.g., plans for stacking objects at the drop off zone), re-stackingplans (e.g., plans for re-stacking at least some of the containers atthe pickup zone), or combinations thereof. The information andinstructions provided by transport plans can be selected based on thearrangement of the containers, the contents of the containers, orcombinations thereof. In some embodiments, the controller 109 caninclude electronic/electrical devices, such as one or more processingunits, processors, storage devices (e.g., external or internal storagedevices, memory, etc.), communication devices (e.g., communicationdevices for wireless or wired connections), and input-output devices(e.g., screens, touchscreen displays, keyboards, keypads, etc.). Exampleelectronic/electrical devices and controller components are discussed inconnection with FIGS. 2 and 6.

The transport unit 106 can transfer the target package 112 (or multipletarget packages 112) from an area associated with the transfer assembly104 to an area associated with the loading unit 108, and the loadingunit 108 can transfer the target package 112 (by, e.g., moving thepallet carrying the target package 112) to a storage location. In someembodiments, the controller 109 can coordinate operation of the transferassembly 104 and the transport unit 106 to efficiently load objects ontostorage shelves.

The robotic system 100 can include other units, such as manipulators,service robots, modular robots, etc., not shown in FIG. 1. For example,in some embodiments, the robotic system 100 can include a de-palletizingunit for transferring the objects from cage carts or pallets ontoconveyors or other pallets, a container-switching unit for transferringthe objects from one container to another, a packaging unit for wrappingthe objects, a sorting unit for grouping objects according to one ormore characteristics thereof, a piece-picking unit for manipulating(e.g., for sorting, grouping, and/or transferring) the objectsdifferently according to one or more characteristics thereof, or acombination thereof. Components and subsystems of the system 100 caninclude different types of end effectors. For example, unloading unit102, transport unit 106, loading unit 108, and other components of therobotic system 100 can also include robotic multi-gripper assemblies.The configurations of the robotic gripper assemblies can be selectedbased on desired carrying capabilities. For illustrative purposes, therobotic system 100 is described in the context of a shipping center;however, it is understood that the robotic system 100 can be configuredto execute tasks in other environments/purposes, such as formanufacturing, assembly, packaging, healthcare, and/or other types ofautomation. Details regarding the task and the associated actions aredescribed below.

Robotic Systems

FIG. 2 is a block diagram illustrating components of the robotic system100 in accordance with one or more embodiments of the presenttechnology. In some embodiments, for example, the robotic system 100(e.g., at one or more of the units or assemblies and/or robots describedabove) can include electronic/electrical devices, such as one or moreprocessors 202, one or more storage devices 204, one or morecommunication devices 206, one or more input-output devices 208, one ormore actuation devices 212, one or more transport motors 214, one ormore sensors 216, or a combination thereof. The various devices can becoupled to each other via wire connections and/or wireless connections.For example, the robotic system 100 can include a bus, such as a systembus, a Peripheral Component Interconnect (PCI) bus or PCI-Express bus, aHyperTransport or industry standard architecture (ISA) bus, a smallcomputer system interface (SCSI) bus, a universal serial bus (USB), anIIC (I2C) bus, or an Institute of Electrical and Electronics Engineers(IEEE) standard 1394 bus (also referred to as “Firewire”). Also, forexample, the robotic system 100 can include bridges, adapters,controllers, or other signal-related devices for providing the wireconnections between the devices. The wireless connections can be basedon, for example, cellular communication protocols (e.g., 3G, 4G, LTE,5G, etc.), wireless local area network (LAN) protocols (e.g., wirelessfidelity (WIFI)), peer-to-peer or device-to-device communicationprotocols (e.g., Bluetooth, Near-Field communication (NFC), etc.),Internet of Things (IoT) protocols (e.g., NB-IoT, Zigbee, Z-wave, LTE-M,etc.), and/or other wireless communication protocols.

The processors 202 can include data processors (e.g., central processingunits (CPUs), special-purpose computers, and/or onboard servers)configured to execute instructions (e.g., software instructions) storedon the storage devices 204 (e.g., computer memory). The processors 202can implement the program instructions to control/interface with otherdevices, thereby causing the robotic system 100 to execute actions,tasks, and/or operations.

The storage devices 204 can include non-transitory computer-readablemediums having stored thereon program instructions (e.g., software).Some examples of the storage devices 204 can include volatile memory(e.g., cache and/or random-access memory (RAM) and/or non-volatilememory (e.g., flash memory and/or magnetic disk drives). Other examplesof the storage devices 204 can include portable memory drives and/orcloud storage devices.

In some embodiments, the storage devices 204 can be used to furtherstore and provide access to master data, processing results, and/orpredetermined data/thresholds. For example, the storage devices 204 canstore master data that includes descriptions of objects (e.g., boxes,cases, containers, and/or products) that may be manipulated by therobotic system 100. In one or more embodiments, the master data caninclude a dimension, a shape (e.g., templates for potential poses and/orcomputer-generated models for recognizing the object in differentposes), mass/weight information, a color scheme, an image,identification information (e.g., bar codes, quick response (QR) codes,logos, etc., and/or expected locations thereof), an expected mass orweight, or a combination thereof for the objects expected to bemanipulated by the robotic system 100. In some embodiments, the masterdata can include manipulation-related information regarding the objects,such as a center-of-mass location on each of the objects, expectedsensor measurements (e.g., force, torque, pressure, and/or contactmeasurements) corresponding to one or more actions/maneuvers, or acombination thereof. The robotic system can look up pressure levels(e.g., vacuum levels, suction levels, etc.), gripping/pickup areas(e.g., areas or banks of vacuum grippers to be activated), and otherstored master data for controlling transfer robots. The storage devices204 can also store object tracking data. In some embodiments, the objecttracking data can include a log of scanned or manipulated objects. Insome embodiments, the object tracking data can include image data (e.g.,a picture, point cloud, live video feed, etc.) of the objects at one ormore locations (e.g., designated pickup or drop locations and/orconveyor belts). In some embodiments, the object tracking data caninclude locations and/or orientations of the objects at the one or morelocations.

The communication devices 206 can include circuits configured tocommunicate with external or remote devices via a network. For example,the communication devices 206 can include receivers, transmitters,modulators/demodulators (modems), signal detectors, signalencoders/decoders, connector ports, network cards, etc. Thecommunication devices 206 can be configured to send, receive, and/orprocess electrical signals according to one or more communicationprotocols (e.g., the Internet Protocol (IP), wireless communicationprotocols, etc.). In some embodiments, the robotic system 100 can usethe communication devices 206 to exchange information between units ofthe robotic system 100 and/or exchange information (e.g., for reporting,data gathering, analyzing, and/or troubleshooting purposes) with systemsor devices external to the robotic system 100.

The input-output devices 208 can include user interface devicesconfigured to communicate information to and/or receive information fromhuman operators. For example, the input-output devices 208 can include adisplay 210 and/or other output devices (e.g., a speaker, a hapticscircuit, or a tactile feedback device, etc.) for communicatinginformation to the human operator. Also, the input-output devices 208can include control or receiving devices, such as a keyboard, a mouse, atouchscreen, a microphone, a user interface (UI) sensor (e.g., a camerafor receiving motion commands), a wearable input device, etc. In someembodiments, the robotic system 100 can use the input-output devices 208to interact with the human operators in executing an action, a task, anoperation, or a combination thereof.

In some embodiments, a controller (e.g., controller 109 of FIG. 1) caninclude the processors 202, storage devices 204, communication devices206, and/or input-output devices 208. The controller can be a standalonecomponent or part of a unit/assembly. For example, each unloading unit,a transfer assembly, a transport unit, and a loading unit of the system100 can include one or more controllers. In some embodiments, a singlecontroller can control multiple units or standalone components.

The robotic system 100 can include physical or structural members (e.g.,robotic manipulator arms) connected at joints for motion (e.g.,rotational and/or translational displacements). The structural membersand the joints can form a kinetic chain configured to manipulate anend-effector (e.g., the gripper) configured to execute one or more tasks(e.g., gripping, spinning, welding, etc.) depending on the use/operationof the robotic system 100. The robotic system 100 can include theactuation devices 212 (e.g., motors, actuators, wires, artificialmuscles, electroactive polymers, etc.) configured to drive or manipulate(e.g., displace and/or reorient) the structural members about or at acorresponding joint. In some embodiments, the robotic system 100 caninclude the transport motors 214 configured to transport thecorresponding units/chassis from place to place. For example, theactuation devices 212 and transport motors can be connected to or partof a robotic arm, a linear slide, or other robotic component.

The sensors 216 can be configured to obtain information used toimplement the tasks, such as for manipulating the structural membersand/or for transporting the robotic units. The sensors 216 can includedevices configured to detect or measure one or more physical propertiesof the robotic system 100 (e.g., a state, a condition, and/or a locationof one or more structural members/joints thereof) and/or for asurrounding environment. Some examples of the sensors 216 can includecontact sensors, proximity sensors, accelerometers, gyroscopes, forcesensors, strain gauges, torque sensors, position encoders, pressuresensors, vacuum sensors, etc.

In some embodiments, for example, the sensors 216 can include one ormore imaging devices 222 (e.g., 2-dimensional and/or 3-dimensionalimaging devices). configured to detect the surrounding environment. Theimaging devices can include cameras (including visual and/or infraredcameras), lidar devices, radar devices, and/or other distance-measuringor detecting devices. The imaging devices 222 can generate arepresentation of the detected environment, such as a digital imageand/or a point cloud, used for implementing machine/computer vision(e.g., for automatic inspection, robot guidance, or other roboticapplications).

Referring now to FIGS. 1 and 2, the robotic system 100 (via, e.g., theprocessors 202) can process image data and/or the point cloud toidentify the target package 112 of FIG. 1, the start location 114 ofFIG. 1, the task location 116 of FIG. 1, a pose of the target package112 of FIG. 1, or a combination thereof. The robotic system 100 can useimage data to determine how to access and pick up objects. Images of theobjects can be analyzed to determine a pickup plan for positioning avacuum gripper assembly to grip targeted objects even though adjacentobjects may also be proximate to the gripper assembly. Imaging outputfrom onboard sensors 216 (e.g., lidar devices) and image data fromremote devices (e.g., the imaging system 160 of FIG. 1) can be utilizedalone or in combination. The robotic system 100 (e.g., via the variousunits) can capture and analyze an image of a designated area (e.g.,inside the truck, inside the container, or a pickup location for objectson the conveyor belt) to identify the target package 112 and the startlocation 114 thereof. Similarly, the robotic system 100 can capture andanalyze an image of another designated area (e.g., a drop location forplacing objects on the conveyor belt, a location for placing objectsinside the container, or a location on the pallet for stacking purposes)to identify the task location 116.

Also, for example, the sensors 216 of FIG. 2 can include positionsensors 224 of FIG. 2 (e.g., position encoders, potentiometers, etc.)configured to detect positions of structural members (e.g., the roboticarms and/or the end-effectors) and/or corresponding joints of therobotic system 100. The robotic system 100 can use the position sensors224 to track locations and/or orientations of the structural membersand/or the joints during execution of the task. The unloading unit,transfer unit, transport unit/assembly, and the loading unit disclosedherein can include the sensors 216.

In some embodiments, the sensors 216 can include contact sensors 226(e.g., force sensors, strain gauges, piezoresistive/piezoelectricsensors, capacitive sensors, elastoresistive sensors, and/or othertactile sensors) configured to measure a characteristic associated witha direct contact between multiple physical structures or surfaces. Thecontact sensors 226 can measure the characteristic that corresponds to agrip of the end-effector (e.g., the gripper) on the target package 112.Accordingly, the contact sensors 226 can output a contact measurementthat represents a quantified measurement (e.g., a measured force,torque, position, etc.) corresponding to physical contact, a degree ofcontact or attachment between the gripper and the target package 112, orother contact characteristics. For example, the contact measurement caninclude one or more force, pressure, or torque readings associated withforces associated with gripping the target package 112 by theend-effector. In some embodiments, the contact measurement can includeboth (1) pressure readings associated with vacuum gripping and (2) forcereadings (e.g., moment readings) associated with carrying object(s).Details regarding the contact measurements are described below.

As described in further detail below, the robotic system 100 (via, e.g.,the processors 202) can implement different actions to accomplish tasksbased on the contact measurement, image data, combinations thereof, etc.For example, the robotic system 100 can regrip the target package 112 ifthe initial contact measurement is below a threshold, such as the vacuumgrip is low (e.g., a suction level is below a vacuum threshold), orcombinations thereof. Also, the robotic system 100 can intentionallydrop the target package 112, adjust the task location 116, adjust aspeed or an acceleration for the action, or a combination thereof basedon one or more transport rules (e.g., if the contact measure or suctionlevel falls below a threshold during execution of the task) and thecontact measurements, image data, and/or other readings or data.

Robotic Transfer Assembly

FIG. 3 illustrates the transfer assembly 104 in accordance with one ormore embodiments of the present technology. The transfer assembly 104can include the imaging system 160 and a robotic arm system 132. Theimaging system 160 can provide image data captured from a targetenvironment with a de-palletizing platform 110. The robotic arm system132 can include a robotic arm assembly 139 and an end effector 140,which includes a vision sensor device 143 and a multi-gripper assembly141 (“gripper assembly 141”). The robotic arm assembly 139 can positionthe end effector 140 above a group of objects in a stack 165 located ata pickup environment 163. The vision sensor device 143 can detect nearbyobjects without contacting, moving, or dislodging objects in the stack165.

Target objects can be secured against the bottom of the end effector140. In some embodiments, the gripper assembly 141 can have addressableregions each selectively capable of drawing in air for providing avacuum grip. In some modes of operation, only addressable regionsproximate to the targeted object(s) draw in air to provide a pressuredifferential directly between the vacuum gripper device and the targetedobject(s). This allows only selected packages (i.e., targeted packages)to be pulled or otherwise secured against the gripper assembly 141 eventhough other gripping portions of the gripper assembly 141 are adjacentto or contact other packages.

FIG. 3 shows the gripper assembly 141 carrying a single object orpackage 112 (“package 112”) positioned above a conveyer 120. The gripperassembly 141 can release the package 112 onto a conveyor belt 120, andthe robotic arm system 132 can then retrieve the packages 112 a, 112 bby positioning the unloaded gripper assembly 141 directly above bothpackages 112 a, 112 b. The gripper assembly 141 can then hold, via avacuum grip, both packages 112 a, 112 b, and the robotic arm system 132can carry the retained packages 112 a, 112 b to a position directlyabove the conveyor 120. The gripper assembly 141 can then release (e.g.,simultaneous or sequentially) the packages 112 a, 112 b onto theconveyor 120. This process can be repeated any number of times to carrythe objects from the stack 165 to the conveyor 120.

The vision sensor device 143 can include one or more optical sensorsconfigured to detect packages held underneath the gripper assembly 141.The vision sensor device 143 can be positioned to the side of thegripper assembly 141 to avoid interference with package pick up/dropoff. In some embodiments, the vision sensor device 143 is movablycoupled to the end effector 140 or robotic arm 139 such that the visionsensor device 143 can be moved to different sides of the gripperassembly 141 to avoid striking objects while detecting a presence of oneor more objects, if any, held by the gripper assembly 141. The position,number, and configurations of the vision sensor devices 143 can beselected based on the configuration of the gripper assembly 141.

With continued reference to FIG. 3, the de-palletizing platform 110 caninclude any platform, surface, and/or structure upon which a pluralityof objects or packages 112 (singularly, “package 112”) may be stackedand/or staged and ready to be transported. The imaging system 160 caninclude one or more imaging devices 161 configured to capture image dataof the packages 112 on the de-palletizing platform 110. The imagingdevices 161 can capture distance data, position data, video, stillimages, lidar data, radar data and/or motion at the pickup environmentor region 163. It should be noted that, although the terms “object” and“package” are used herein, the terms include any other items capable ofbeing gripped, lifted, transported, and delivered such as, but notlimited to, “case,” “box”, “carton,” or any combination thereof.Moreover, although polygonal boxes (e.g., rectangular boxes) areillustrated in the drawings disclosed herein, the shapes of the boxesare not limited to such shape but includes any regular or irregularshape that, as discussed in detail below, is capable of being gripped,lifted, transported, and delivered.

Like the de-palletizing platform 110, the receiving conveyor 120 caninclude any platform, surface, and/or structure designated to receivethe packages 112 for further tasks/operations. In some embodiments, thereceiving conveyor 120 can include a conveyor system for transportingthe package 112 from one location (e.g., a release point) to anotherlocation for further operations (e.g., sorting and/or storage).

FIG. 4 is a front view of the end effector 140 coupled to the roboticarm 139 in accordance with some embodiments of the present technology.FIG. 5 is a bottom view of the end effector 140 of FIG. 4. The visionsensor device 143 can include one or more sensors 145 configured todetect packages and a calibration board 147 used to, for example,calibrate the position of the gripper assembly 141 relative to thevision sensor device 143. In some embodiments, the calibration board 147can be a placard with a pattern or design used for calibrating ordefining the position of the end effector 140 or gripper assembly 141within the operating environment, position of the robotic arm 139, or acombination thereof. The gripper assembly 141 can include addressablevacuum zones or regions 117 a, 117 b, 117 c (collectively “vacuumregions 117”) defining a gripping zone 125. The description of onevacuum region 117 applies to the other vacuum regions 117 unlessindicated otherwise. In some embodiments, each vacuum region 117 can bea suction channel bank that includes components connected to a vacuumsource external to the end effector 140. The vacuum regions 117 caninclude gripping interfaces 121 (one identified in FIG. 4) against whichobjects can be held.

Referring now to FIG. 4, the vacuum region 117 a can draw in air to holdthe package 112 and can reduce or stop drawing in air to release thepackage 112. The vacuum regions 117 b, 117 c (illustrated not holdingpackages) can independently draw in air (indicated by arrows) to holdpackages at corresponding positions 113 a, 113 b (illustrated in phantomline in FIG. 4). Referring now to FIG. 5, the vacuum regions 117 caninclude a group or bank of suction elements 151 (one identified in FIG.5) through which air is drawn. The suction elements 151 can beevenly/uniformly or unevenly spaced apart from one another and can bearranged in a desired pattern (e.g., an irregular or regular pattern).The vacuum regions 117 can have the same or different number,configurations, and/or pattern of suction elements 151. To carry apackage that matches the geometry of the vacuum region 117, air can bedrawn through each suction element 151 of the vacuum region 117. Tocarry smaller packages, air can be drawn through a subset of the suctionelements 151 matching the geometry of the package (e.g., suctionelements 151 positioned within the boundary or perimeter of thepackage). For example, air can be drawn through a subset of the suctionelements for one of the vacuum region 117, such as only the suctionelements 151 immediately adjacent to or overlying a target surface to begripped. As shown in FIG. 5, for example, the suction elements 151within a boundary 119 (illustrated in dashed line) can be used to grip acorresponding circular surface of a package.

When all of the vacuum regions 117 are active, the end effector 140 canprovide a generally uniform gripping force along the each of thegripping interfaces 121 or entire bottom surface 223. In someembodiments, the bottom surface 223 is a generally continuous andsubstantially uninterrupted surface and the distance or pitch betweensuction elements 151 of adjacent vacuum regions 117 can be less than,equal to, or greater than (e.g., 2×, 3×, 4×, etc.) the pitch betweensuction elements 151 of the same vacuum region 117. The end effector 140can be configured to hold or affix object(s) via attractive forces, suchas achieved by forming and maintaining a vacuum condition between thevacuum regions 117 and the object. For example, the end effector 140 caninclude one or more vacuum regions 117 configured to contact a surfaceof the target object and form/retain the vacuum condition in the spacesbetween the vacuum regions 117 and the surface. The vacuum condition canbe created when the end effector 140 is lowered via the robotic arm 139,thereby pressing the vacuum regions 117 against the surface of thetarget object and pushing out or otherwise removing gases between theopposing surfaces. When the robotic arm 139 lifts the end effector 140,a difference in pressure between the spaces inside the vacuum regions117 and the surrounding environment can keep the target object attachedto the vacuum regions 117. In some embodiments, the air-flow ratethrough the vacuum regions 117 of the end effector 140 can bedynamically adjusted or based on the contact area between the targetobject and a contact or gripping surface of the vacuum regions 117 toensure that a sufficient grip is achieved to securely grip the targetobject. Similarly, the air-flow rate thought the vacuum regions 117 canbe adjusted dynamically to accommodate the weight of the target object,such as increasing the air flow for heavier objects, to ensure thatsufficient grip is achieved to securely grip the target object. Examplesuction elements are discussed in connection with FIG. 15.

FIG. 6 is a functional block diagram of the transfer assembly 104 inaccordance with one or more embodiments of the present technology. Aprocessing unit 150 (PU) can control the movements and/or other actionsof the robotic arm system 132. The PU 150 can receive image data fromsensors (e.g., sensors 161 of the imaging system 160 of FIG. 3), sensors145 of the vision sensor device 143, or other sensors or detectorscapable of collecting image data, including video, still images, lidardata, radar data, or combinations thereof. In some embodiments, theimage data can be indicative or representative of a surface image (SI)of the package 112.

The PU 150 can include any electronic data processing unit whichexecutes software or computer instruction code that could be stored,permanently or temporarily, in memory 152, a digital memory storagedevice or a non-transitory computer-readable media including, but notlimited to, random access memory (RAM), disc drives, magnetic memory,read-only memory (ROM), compact disc (CD), solid-state memory, securedigital cards, and/or compact flash cards. The PU 150 may be driven bythe execution of software or computer instruction code containingalgorithms developed for the specific functions embodied herein. In someembodiments, the PU 150 may be an application-specific integratedcircuit (ASIC) customized for the embodiments disclosed herein. In someembodiments, the PU 150 can include one or more of microprocessors,Digital Signal Processors (DSPs), Programmable Logic Devices (PLDs),Programmable Gate Arrays (PGAs), and signal generators; however, for theembodiments herein, the term “processor” is not limited to such exampleprocessing units and its meaning is not intended to be construednarrowly. For instance, the PU 150 can also include more than oneelectronic data processing unit. In some embodiments, the PU 150 couldbe a processor(s) used by or in conjunction with any other system of therobotic system 100 including, but not limited to, the robotic arm system130, the end effector 140, and/or the imaging system 160. The PU 150 ofFIG. 6 and the processor 202 of FIG. 2 can be the same component ordifferent components.

The PU 150 may be electronically coupled (via, e.g., wires, buses,and/or wireless connections) to systems and/or sources to facilitate thereceipt of input data. In some embodiments, operatively coupled may beconsidered as interchangeable with electronically coupled. It is notnecessary that a direct connection be made; instead, such receipt ofinput data and the providing of output data could be provided through abus, through a wireless network, or as a signal received and/ortransmitted by the PU 150 via a physical or a virtual computer port. ThePU 150 may be programmed or configured to execute the methods discussedherein. In some embodiments, the PU 150 may be programmed or configuredto receive data from various systems and/or units including, but notlimited to, the imaging system 160, end effector 140, etc. In someembodiments, the PU 150 may be programmed or configured to provideoutput data to various systems and/or units.

The imaging system 160 could include one or more sensors 161 configuredto capture image data representative of the packages (e.g., packages 112located on the de-palletizing platform 110 of FIG. 3). In someembodiments, the image data can represent visual designs and/or markingsappearing on one or more surfaces of the package from which adetermination of a registration status of the package may be made. Insome embodiments, the sensors 161 are cameras configured to work withina targeted (e.g., visible and/or infrared) electromagnetic spectrumbandwidth and used to detect light/energy within the correspondingspectrum. In some camera embodiments, the image data is a set of datapoints forming point cloud, the depth map, or a combination thereofcaptured from one or more three-dimensional (3D) cameras and/or one ormore two-dimensional (2-D) cameras. From these cameras, distances ordepths between the imaging system 160 and one or more exposed (e.g.,relative to a line of sight for the imaging system 160) surfaces of thepackages 112 may be determined. In some embodiments, the distances ordepths can be determined by using an image recognition algorithm(s),such as contextual image classification algorithm(s) and/or edgedetection algorithm(s). Once determined, the distance/depth values maybe used to manipulate the packages via the robotic arm system. Forexample, the PU 150 and/or the robotic arm system can use thedistance/depth values for calculating the position from where thepackage may be lifted and/or gripped. It should be noted that datadescribed herein, such as the image data, can include any analog ordigital signal, either discrete or continuous, which could containinformation or be indicative of information.

The imaging system 160 can include at least one display unit 164configured to present operational information (e.g., status information,settings, etc.), an image of the package(s) 112 captured by the sensors162, or other information/output that may be viewed by one or moreoperators of the robotic system 100 as discussed in detail below. Inaddition, the display units 164 can be configured to present otherinformation such as, but not limited to, symbology representative oftargeted packages, non-targeted packages, registered packages, and/orunregistered instances of the packages.

The vision sensor device 143 can communicate with the PU 150 via wireand/or wireless connections. The vision sensor 145 can be video sensors,CCD sensors, lidar sensors, radar sensors, distance-measuring ordetecting devices, or the like. Output from the vision sensor device 143can be used to generate a representation of the package(s), such as adigital image and/or a point cloud, used for implementingmachine/computer vision (e.g., for automatic inspection, robot guidance,or other robotic applications). The field of view (e.g., 30 degrees, 90degrees, 120 degrees, 150 degrees, 180 degrees, 210 degrees, 270 degreesof horizontal and/or vertical FOV) and the range capability of thevision sensor device 143 can be selected based on the configuration ofthe gripper assembly 141. (FIG. 4 shows an exemplary horizontal FOV ofabout 90 degrees.) In some embodiments, the vision sensors 145 are lidarsensors with one or more light sources (e.g., lasers, infrared lasers,etc.) and optical detectors. The optical detectors can detect lightemitted by the light sources and reflected by surfaces of packages. Thepresence and/or distance to packages can be determined based on thedetected light. In some embodiments, the sensors 145 can scan an area,such as substantially all of a vacuum gripping zone (e.g., vacuumgripping zone 125 of FIG. 4). For example, the sensors 154 can includeone or more deflectors that move to deflect emitted light across adetection zone. In some embodiments, the sensors 154 are scanninglaser-based lidar sensors capable of scanning vertically and/orhorizontally, such as a 10° lidar scan, a 30° lidar scan, a 50° lidarscan, etc.). The configuration, FOV, sensitivity, and output of thesensors 145 can be selected based on the desired detection capabilities.In some embodiments, the sensors 145 can include both presence/distancedetectors (e.g., radar sensors, lidar sensor, etc.) and one or morecameras, such as three-dimensional or two-dimensional cameras. Distancesor depths between the sensors and one or more surfaces of packages canbe determined using, for example, one or more image recognitionalgorithms. The display unit 147 can be used to view image data, viewsensor status, perform calibration routines, view logs and/or reports,or other information or data, such as, but not limited to, symbologyrepresentative of targeted, non-targeted, registered, and/orunregistered instances of packages 112.

To control the robotic system 100, the PU 150 can use output from one orboth the sensors 145 and sensors 161. In some embodiments, image outputfrom sensors 161 is used to determine an overall transfer plan,including an order for transporting objects. Image output from thesensors 145, as well as sensors 205 (e.g., a force detector assembly),can be used to position a multi-gripping assembly with respect toobjects, confirm object pickup, and monitor transport steps.

With continued reference to FIG. 6, the RDS 170 could include anydatabase and/or memory storage device (e.g., a non-transitorycomputer-readable media) configured to store the registration records172 for a plurality of the packages 112, data 173 for vacuum grippers.For example, the RDS 170 can include read-only memory (ROM), compactdisc (CD), solid-state memory, secure digital cards, compact flashcards, and/or data storage servers or remote storage devices.

In some embodiments, the registration records 172 can each includephysical characteristics or attributes for the corresponding package112. For example, each registration record 172 can include, but is notbe limited to, one or more template SIs, vision data (e.g., referenceradar data, reference lidar data, etc.), 2-D or 3-D size measurements, aweight, and/or center of mass (CoM) information. The template SIs canrepresent known or previously determined visible characteristics of thepackage including the design, marking, appearance, exteriorshape/outline, or a combination thereof of the package. The 2-D or 3-Dsize measurements can include lengths, widths, heights, or combinationthereof for the known/expected packages.

In some embodiments, the RDS 170 can be configured to receive a newinstance of the registration record 172 (e.g., for a previously unknownpackage and/or a previously unknown aspect of a package) created inaccordance with the embodiments disclosed below. Accordingly, therobotic system 100 can automate the process for registering the packages112 by expanding the number of registration records 172 stored in theRDS 170, thereby making a de-palletizing operation more efficient withfewer unregistered instances of the packages 112. By dynamically (e.g.,during operation/deployment) updating the registration records 172 inthe RDS 170 using live/operational data, the robotic system 100 canefficiently implement a computer-learning process that can account forpreviously unknown or unexpected conditions (e.g., lighting conditions,unknown orientations, and/or stacking inconsistencies) and/or newlyencountered packages. Accordingly, the robotic system 100 can reduce thefailures resulting from “unknown” conditions/packages, associated humanoperator interventions, and/or associated task failures (e.g., lostpackages and/or collisions).

The RDS 170 can include vacuum gripper data 173, including, but notlimited to, characteristics or attributes, including the number ofaddressable vacuum regions, carrying capability of a vacuum gripperdevice (e.g., multi-gripper assembly), vacuum protocols (e.g., vacuumlevels, airflow rates, etc.), or other data used to control the roboticarm system 130 and/or end effector 140. An operator can inputinformation about the vacuum gripper installed in the robotic arm system130. The RDS 170 then identifies vacuum gripper data 173 correspondingto the vacuum gripper device for operation. In some embodiments, thevacuum gripper device (e.g., gripper assembly 141 of FIG. 3) isautomatically detected by the robotic arm 139, and the RDS 170 is usedto identify information about the detected vacuum gripper device. Theidentified information can be used to determine settings of the vacuumgripper device. Accordingly, different vacuum gripper devices ormulti-gripper assemblies can be installed and used with the robotic armsystem 130.

End Effectors

FIG. 7 is a front, top isometric view of a portion of the end effector140 in accordance with one or more embodiments of the presenttechnology. FIG. 8 is a front, bottom isometric view of the end effector140 of FIG. 7. Referring now to FIG. 7, the end effector 140 can includea mounting interface or bracket 209 (“mounting bracket 209”) and a forcedetector assembly 205 coupled to the bracket 209 and the gripperassembly 141. A fluid line 207 can be fluidically coupled to apressurization device, such as a vacuum source 221 (not shown in FIG. 8)and the gripper assembly 141.

The FOV (a variable or a fixed FOV) of the vision sensor device 143 isdirected generally underneath the gripper assembly 141 to providedetection of any objects carried underneath the gripper assembly 141.The vision sensor device 143 can be positioned along the perimeter ofthe end effector 140 such that the vision sensor device 143 is below thesubstantially horizontal plane of one or more of the vacuum regions 117(one identified), and more specifically, the gripping surface of thegripping interface 121 (one identified). The term “substantiallyhorizontal” generally refers to an angle within about +/−2 degrees ofhorizontal, for example, within about +/−1 degree of horizontal, such aswithin about +/−0.7 degrees of horizontal. In general, the end effector140 includes multiple vacuum regions 117 that enable the robotic system100 to grip the target objects that otherwise would not be grippable bya single instance of the vacuum regions 117. However, a larger area willbe obscured from detection sensors due to the larger size of the endeffector 140 relative to the end effector 140 with the single instanceof vacuum regions 117. As one advantage, the vision sensor device 143positioned below the horizontal plane of the gripping interface 121 canprovide the vision sensor device 143 with a FOV that includes thegripping interface 121 during contact initiation with objects, includingthe target object, that would normally be obscured for other instancesof the vision sensor device 143 that are not attached to the endeffector 140 or positioned in different locations within the operatingenvironment of the robotic system 100. As such, the unobscured FOV canprovide the robotic system with real-time imaging sensor informationduring the gripping operations, which can enable real-time or on the flyadjustments to the position and motion of the end effector 140. As afurther advantage, the proximity between the vision sensor device 143positioned below the horizontal plane of the gripping interface 121 andobjects (e.g., non-targeted objects 112 a, 112 b of FIG. 3) increasesthe precision and accuracy during the gripping operation, which canprotect or prevent damage to the target object 112 and the non-targetedobjects adjacent to the target object 112,a 112 b from the end-effector140, such as by crushing of the objects.

For illustrative purposes, the vision sensor device 143 can bepositioned at a corner of the end-effector 140 along the effector width,however, it is understood that the vision sensor device 143 can bepositioned differently. For example, the vision sensor device 143 can bepositioned at the center of the width or length of the end-effector 140.As another example, the vision sensor device 143 can be positioned atanother corner or other positions along the effector length.

The vacuum source 221 (FIG. 7) can include, without limitation, one ormore pressurization devices, pumps, valves, or other types of devicescapable of providing a negative pressure, drawing a vacuum (includingpartial vacuum), or creating a pressure differential. In someembodiments, air pressure can either be controlled with one or moreregulators, such as a regulator between the vacuum source 221 and thegripper assembly 141 or a regulator in the gripper assembly 141. Whenthe vacuum source 221 draws a vacuum, air can be drawn (indicated byarrows in FIG. 8) into the bottom 224 of the gripper assembly 141. Thepressure level can be selected based on the size and weight of theobjects to be carried. If the vacuum level is too low, the gripperassembly 141 may not be able to pick up the target object(s). If thevacuum level is too high, the outside of the package could be damaged(e.g., a package with an outer plastic bag could be torn due to a highvacuum level). According to some embodiments, the vacuum source 221 canprovide vacuum levels of approximately 100 mBar, 500 mBar, 1,000 mBar,2,000 mBar, 4,000 mBar, 6,000 mBar, 8,000 mBar, or the like. Inalternative embodiments, higher or lower vacuum levels are provided. Insome embodiments, the vacuum level can be selected based on the desiredgripping force. The vacuum gripping force of each region 117 can beequal to or greater than about 50N, 100N, 150N, 200N, or 300N at avacuum level (e.g., 25%, 50%, or 75% maximum vacuum level, i.e., maximumvacuum level for the vacuum source 221). These gripping forces can beachieved when picking up a cardboard box, plastic bag, or other suitablepackage for transport. Different vacuum levels can be used, includingwhen transporting the same object or different objects. For example, arelatively high vacuum can be provided to initially grip the object.Once the package has been gripped, the gripping force (and therefore thevacuum level) required to continue to hold the object can be reduced, soa lower vacuum level can be provided. The gripping vacuum can beincreased to maintain a secure grip when performing certain tasks.

The force detector assembly 205 can include one or more sensors 203 (oneillustrated) configured to detect forces indicative of the load carriedby the end effector 140. The detected measurements can include linearforces measurements along an axis and/or axes of a coordinate system,moment measurements, pressures measurements, or combinations thereof. Insome embodiments, the sensor 203 can be a F-T sensor that includes acomponent with six-axis force sensors configured to detect up to threeaxis forces (e.g., forces detected along x-, y-, and z-axes of aCartesian coordinate system) and/or three axis moments (e.g., momentsdetected about x-, y-, and z-axes of the Cartesian coordinate system).In some embodiments, the sensor 203 could include a built-in amplifierand microcomputer for signal processing, an ability to make static anddynamic measurements, and/or an ability to detect instant changes basedon a sampling interval. In some embodiments with reference made to theCartesian coordinate system, force measurement(s) along one or more axis(i.e., F(x-axis), F(y-axis), and/or F(z-axis)) and/or momentmeasurement(s) about one or more axis (i.e., M(x-axis), M(y-axis),and/or M(z-axis)) may be captured via the sensor 203. By applying CoMcalculation algorithms, the weight of the packages, positions ofpackages, and/or number of packages can be determined. For example, theweight of the packages may be computed as a function of the forcemeasurement(s), and the CoM of the package may be computed as a functionof the force measurement(s) and the moment measurement(s). In someembodiments, the weight of the packages is computed as a function of theforce measurement(s), package position information from the visionsensor device 143, and/or gripping information (e.g., locations at whicha seal with the package(s) is achieved). In some embodiments, thesensors 203 could be communicatively coupled with a processing unit(e.g., PU 150 of FIG. 6) via wired and/or wireless communications.

In some embodiments, output readings from both the force detectorassembly 205 and the vision sensor device 143 can be used. For example,relative positions of objects can be determined based on output from thevision sensor device 143. The output from the force detector assembly205 can then be used to determine information about each object, such asthe weight/mass of each object. The force detector assembly 205 caninclude contact sensors, pressure sensors, force sensors, strain gauges,piezoresistive/piezoelectric sensors, capacitive sensors,elastoresistive sensors, torque sensors, linear force sensors, or othertactile sensors, configured to measure a characteristic associated witha direct contact between multiple physical structures or surfaces. Forexample, the force detector assembly 205 can measure the characteristicthat corresponds to a grip of the end-effector on the target object ormeasure the weight of the target object. Accordingly, the force detectorassembly 205 can output a contact measure that represents a quantifiedmeasure, such as a measured force or torque, corresponding to a degreeof contact or attachment between the gripper and the target object. Forexample, the contact measure can include one or more force or torquereadings associated with forces applied to the target object by theend-effector. The output can be from the force detector assembly 205 orother detectors that are integrated with or attached to the end effector140. For example, the sensor information from the contact sensors, suchas weight or weight distribution of the target object based on the forcetorque sensor information, in combination with the imaging sensorinformation, such as dimension of the target object, can be used by therobotic system to determine the identity of the target object, such asby an auto-registration or automated object registration system.

FIG. 9 is an exploded isometric view of the gripper assembly 141 inaccordance with one or more embodiments of the present technology. Thegripper assembly 141 includes a housing 260 and an internal assembly263. The housing 260 can surround and protect the internal componentsand can define an opening 270 configured to receive at least a portionof the force detector assembly 205. The internal assembly 263 caninclude a gripper bracket assembly 261 (“bracket assembly 261”), amanifold assembly 262, and a plurality of grippers 264 a, 264 b, 264 c(collectively “grippers 264”). The bracket assembly 261 can hold each ofthe vacuum grippers 264, which can be fluidically coupled in series orparallel to a fluid line (e.g., fluid line 207 of FIG. 7) via themanifold assembly 262, as discussed in connection with FIGS. 10 and 11.In some embodiments, the bracket assembly 261 includes an elongatedsupport 269 and brackets 267 (one identified) connecting the grippers264 to the elongated support 269. The gripper assembly 141 can includesuction elements, sealing members (e.g., sealing panels), and othercomponents discussed in connection with FIGS. 13-15.

FIGS. 10 and 11 are a rear, top isometric view and a plan view,respectively, of components of the gripper assembly in accordance withone or more embodiments of the present technology. The manifold assembly262 can include gripper manifolds 274 a, 274 b, 274 c (collectively“manifolds 274”) coupled to respective grippers 264 a, 264 b, 264 c. Forexample, the manifold 274 a controls air flow associated with thegripper 264 a. In some embodiments, the manifolds 274 can be connectedin parallel or series to a pressurization source, such as the vacuumsource 221 of FIG. 7. In other embodiments, each manifold 274 can befluidly coupled to an individual pressurization device.

The manifolds 274 can be operated to distribute the vacuum to one, some,or all of the grippers 264. For example, the manifold 274 a can be in anopen state to allow air to flow through the bottom of the gripper 264 a.The air flows through the manifold 274 a, and exits the vacuum gripperassembly via a line, such as the line 207 of FIG. 7. The other manifolds274 b, 274 c can be in a closed state to prevent suction at themanifolds 274 b, 274 c. Each manifold 274 a can include, withoutlimitation, one or more lines connected to each of the suction elements.In other embodiments, the suction elements of the gripper 264 a areconnected to an internal vacuum chamber. The gripper manifolds 274 caninclude, without limitation, one or more lines or passages, valves(e.g., check valves, globe valves, three-way valves, etc.), pneumaticcylinders, regulators, orifices, sensors, and/or other componentscapable of controlling the flow of fluid. Each manifold 274 can be usedto distribute suction evenly or unevenly to suction elements or groupsof suction elements to produce uniform or nonuniform vacuum grippingforces. An electronics line can communicatively couple the manifolds 274to a controller to provide power to and control over components of themodules and components thereof. In one embodiment, individual manifolds274 can include common interfaces and plugs for use with commoninterfaces and plugs, which may make it possible to add and removemanifolds 274 and components quickly and easily, thereby facilitatingsystem reconfiguration, maintenance, and/or repair.

The number, arrangement, and configuration of the grippers can beselected based on a desired number of addressable vacuum regions. FIG.12 is an isometric view of internal components of a vacuum gripperassembly 300 (housing not shown) suitable for use with the environmentof FIGS. 1-2 and the transfer assembly 141 of FIGS. 3-6 in accordancewith one or more embodiments of the present technology. The vacuumgripper assembly 300 can include six vacuum grippers 302 (oneidentified) in a generally rectangular arrangement. In otherembodiments, the grippers can be in a circular arrangement, squarearrangement, or other suitable arrangement and can have similar ordifferent configurations. The grippers can have other shapes including,without limitation, oval shapes, non-polygonal shapes, or the like. Thegrippers can include suction elements (e.g., suction tubes, suctioncups, sealing member, etc.), sealing member, valve plates, grippermechanisms, and other fluidic components for providing grippingcapability.

One or more sensors, vision sensor devices, and other componentdiscussed in connection with FIGS. 1-11 can be incorporated into or usedwith the vacuum gripper assembly 300. Suction elements, sealing member,and other components are discussed in connection with FIGS. 13-15.

The vacuum grippers can be arranged in series. For example, vacuumgrippers can be arranged one next to another in a I×3 configuration,which provides two lateral gripping position and one central grippingposition. However, it is understood that the end effectors can include adifferent number of the vacuum grippers, suction channel banks, orvacuum regions in different configurations relative to one another. Forexample, the end effector can include four of the vacuum grippers orsuction channel banks arranged in a 2×2 configuration. The vacuumregions can have a width dimension that is the same or similar to thelength dimension to have a symmetric square shape. As another example,the end effector can include a different number of the vacuum regions,such as two of vacuum regions or more than three of vacuum regionshaving the same or different length dimension and/or width dimensionform one another. In yet a further example, the vacuum grippers can bearranged in various configurations, such as a 2×2 configuration withfour of the vacuum regions, a 1:2:2 configuration that includes five ofthe vacuum grippers, or other geometric arrangements and/orconfigurations.

FIG. 13 shows a multi-gripper assembly 400 (“gripper assembly 400”)suitable for use with robotic systems (e.g., robotic system 100 of FIGS.1-2) in accordance with some embodiments of the present technology. FIG.14 is an exploded view of the gripper assembly 400 of FIG. 13. Thegripper assembly 400 can be any gripper or gripper assembly configuredto grip a package from a stationary position (e.g., a stationaryposition on a de-palletizing platform such as a platform 110 of FIG. 3).The gripper assembly device 400 can include a gripper mechanism 410 anda contact or sealing member 412 (“sealing member 412”). The grippermechanism 410 includes a main body 414 and a plurality of suctionelements 416 (one identified in FIG. 14) each configured to pass throughan opening 418 (one identified in FIG. 14) of the member 412. Whenassembled, each of the suction elements 416 can extend through, eitherpartially or completely, a corresponding opening 418. For example, thesuction elements 416 can extend through a first side 419 toward thesecond side 421 of the sealing member 412.

FIG. 15 is a partial cross-sectional view of the sealing member 412 andthe suction element 416. The suction element 416 can be in fluidcommunication with a line (e.g., line 422 of FIG. 14) via a vacuumchamber and/or internal conduit 430. A valve 437 (e.g., check valve,relief valve, etc.) can be positioned along an air flow path 436. Asensor 434 can be positioned to detect a vacuum level and can be incommunication, via a wired or wireless connection, with a controller(e.g., controller 109 of FIG. 1) or processing unit (e.g., processingunit 150 of FIG. 6). A lower end 440 of the suction element 416 caninclude, without limitation, a suction cup or another suitable featurefor forming a desired seal (e.g., a generally airtight seal or othersuitable seal) with an object's surface. When the lower end 440 isproximate to or contacts the object, the object can be pulled againstthe sealing member 412 when air is drawn into a port/inlet 432 (“inlet432”) of the suction element 416 (as indicated by arrows). The air flowsupwardly along a flow path 426 and through a passageway 433 of thesuction element 416. The air can flow through a valve 437 and into theconduit 430. In some embodiments, the conduit 430 can be connected to avacuum chamber 439. For example, some or all of the suction elements 416can be connected to the vacuum chamber 439. In other embodiments,different groups of suction elements 416 can be in fluid communicationwith different vacuum chambers. The suction elements 416 can have anundulating or bellowed configuration, as shown, to allow axialcompression without constricting the airflow passageway 433 therein. Theconfigurations, heights, and dimensions of the suction elements 416 canbe selected based on the desired amount of compressibility.

The sealing member 412 can be made, in whole or part, of compressiblematerials configured to deform to accommodate surfaces with differentgeometries, including highly contoured surfaces. The sealing member 412can be made, in whole or in part, of foam, including closed-cell foam(e.g., foam rubber). The material of the sealing member 412 can beporous to allow small amounts of air flow (i.e., air leakage) to avoidapplying high negative pressures that could, for example, damagepackaging, such as plastic bags.

Operational Flow

FIG. 16 is a flow diagram of a method 490 for operating a robotic systemin accordance with one or more embodiments of the present disclosure. Ingeneral, a transport robot can receive image data representative of atleast a portion of a pickup environment. The robot system can identifytarget objects based on the received image data. The robot system canuse a vacuum gripper assembly to hold onto the identified targetobject(s). Different units, assemblies, and subassemblies of the robotsystems 100 of FIG. 1 can perform the method 490. Details of the method490 are discussed in detail below.

At block 500, the robotic system 100 can receive image datarepresentative of at least a portion of an environment. For example, thereceived image data can be representative of at least a portion of thestack 165 at the pickup environment 163 of FIG. 3. The image data caninclude, without limitation, video, still images, lidar data, radardata, bar code data, or combinations thereof. In some embodiments, forexample, the sensors 161 of FIG. 3 can capture video or still imagesthat are transmitted (e.g., via a wired or wireless connection) to acomputer or controller, such as the controller 109 of FIGS. 1 and 6.

At block 502, the computer 109 (FIG. 1) can analyze image data toidentify target objects in a group of objects, a stack of objects, etc.For example, the controller 109 can identify individual objects based onthe received image data and surface images/data stored by the RDS 170(FIG. 6). In some embodiments, information from the drop off location isused to select the target object. For example, a target object can beselected based on the amount of available space at the drop offlocation, preferred stacking arrangement, etc. A user can inputselection criteria for determining the order of object pick up. In someembodiments, a mapping of the pickup environment (e.g., pickupenvironment 163 of FIG. 3) can be generated based on the received imagedata. In some mapping protocols, edge detection algorithms are used toidentify edges of objects, surfaces, etc. The mapping can be analyzed todetermine which objects at the pickup region are capable of beingtransported together. In some embodiments, a group of objects capable ofbeing simultaneously lifted and carried by the vacuum gripper areidentified as targeted objects.

The robotic system 100 of FIG. 1 can select the target package or object112 from source objects as the target of a task to be performed. Forexample, the robotic system 100 can select the target object to bepicked up according to a predetermined sequence, set of rules, templatesof object outlines, or a combination thereof. As a specific example, therobotic system 100 can select the target package as an instance of thesource packages that are accessible to the end effector 140, such as aninstances of the source packages 112 located on top of a stack of thesource packages, according to the point cloud/depth map representing thedistances and positions relative to a known location of the imagedevices. In another specific example, the robotic system 100 can selectthe target object as an instance of the source packages 112 located at acorner or edge and having two or more surfaces that are exposed to oraccessible to the end effector 140. In a further specific example, therobotic system 100 can select the target object according to apredetermined pattern, such as left to right or nearest to furthestrelative to a reference location, without or minimally disturbing ordisplacing other instances of the source packages.

At block 504, the controller 109 can select the vacuum grippers orregions for gripping the target objects. For example, the controller 109(FIG. 1) can select the vacuum region 117 a (FIG. 4) for gripping thepackage 112, illustrated in FIG. 3, because substantially the entirepackage 112 (i.e., target object) is directly beneath the vacuum region117 a. A vacuum can be drawn through substantially all of the suctionelements 151 (e.g., at least 90%, 95%, 98% of the suction elements 151)of the vacuum region 117 a of FIG. 4.

At block 506, the controller 109 generates one or more commands forcontrolling the robotic system 100. In some modes of operation, thecommands can cause the robotic system to suck in air at the identifiedor selected addressable vacuum regions. For example, the controller 109can generate one or more pickup commands to cause a vacuum source (e.g.,vacuum source 221 of FIG. 7) to provide a vacuum at a selected vacuumlevel. The vacuum level can be selected based on the weight or mass ofthe target object(s), tasks to be performed, etc. Commands can be sentto the gripper assembly 141 to cause the manifold 262 to operate toprovide suction at the selected regions or grippers. Feedback from thevision sensor device 143 (FIG. 7) can be used to monitor the pickup andtransfer process.

At block 508, the vision sensor device 143 can be used to verify theposition of the end effector 140 relative to objects, including sourceor target objects, such as the packages 112 of FIG. 1. The vision sensordevice 143 can be used to continuously or periodically monitor therelative position of the end effector 140 relative to objects before andduring object pickup, during object transport, and/or during and afterobject drop off. The output from vision sensor device 143 can also beused to count objects, (e.g., count the number of target or sourceobjects) or otherwise analyze objects, including analyzing stacks ofobjects. The vision sensor device 143 can also be used to obtainenvironmental information used to navigate the robotic system 100.

At block 510, the controller 109 generates command to cause actuationdevices (e.g., actuation devices 212), motors, servos, actuators, andother components of the robotic arm 139 to move the gripper assembly141. Transfer commands can be generated by the robotic system to causethe robotic transport arm to robotically move the gripper assembly 141carrying the objects between locations. The transport commands can begenerated based on a transport plan that includes a transport path todeliver the object to a drop off location without causing the object tostrike another object. The vision sensor device 143 (FIG. 7) can be usedto avoid collisions.

The method 490 can be performed to grip multiple target objects. The endeffector 140 can be configured to grip multiple instances of the targetpackage or object from among the source packages or objects. Forexample, the robotic system 100 can generate instructions for the endeffector 140 to engage multiple instances of the vacuum regions 117 toperform the gripping operation to simultaneously grip multiple instancesof the target object. As a specific example, the end effector 140 can beused to execute instructions for the gripping operation of grippingmultiple instances of the target object separately and in sequence, oneafter the other. For instance, the instructions can include performingthe gripping operation using one of the vacuum regions 117 to grip afirst instance of the target object 112 that is in one pose or oneorientation, then, if necessary, repositioning the end effector 140 toengage a second or different instance of the vacuum regions 117 to gripa second instance of the target object. In another specific example, theend effector 140 can be used to execute instructions for the grippingoperation of simultaneous gripping of separate instances of the targetobject. For instance, the end effector 140 can be positioned tosimultaneously contact two or more instances of the target object andengage each of the corresponding instances of vacuum regions 117 toperform the gripping operation on each of the multiple instances of thetarget object. In the above embodiments, each of the vacuum regions 117can be independently operated as necessary to perform the differentgripping operations.

FIG. 17 is a flow diagram of a method 700 for operating the roboticsystem 100 of FIG. 1 according to a base plan in accordance with one ormore embodiments of the present technology. The method 700 includessteps that can be incorporated into the method 490 of FIG. 16 and can beimplemented based on executing the instructions stored on one or more ofthe storage devices 204 of FIG. 2 with one or more of the processors 202of FIG. 2 or the controller 109 of FIG. 6. Data captured by the visionsensor devices and sensor output can be used at various steps of themethod 700 as detailed below.

At block 702, the robotic system 100 can interrogate (e.g., scan) one ormore designated areas, such as the pickup area and/or the drop area(e.g., a source drop area, a destination drop area, and/or a transitdrop area). In some embodiments, the robotic system 100 can use (via,e.g., commands/prompts sent by the processors 202 of FIG. 2) one or moreof the imaging devices 222 of FIG. 2, sensors 161 and/or 145 of FIG. 6,or other sensors to generate imaging results of the one or moredesignated areas. The imaging results can include, without limitation,captured digital images and/or point clouds, object position data, orthe like.

At block 704, the robotic system 100 can identify the target package 112of FIG. 1 and associated locations (e.g., the start location 114 of FIG.1 and/or the task location 116 of FIG. 1). In some embodiments, forexample, the robotic system 100 (via, e.g., the processors 202) cananalyze the imaging results according to a pattern recognition mechanismand/or a set of rules to identify object outlines (e.g., perimeter edgesor surfaces). The robotic system 100 can further identify groupings ofobject outlines (e.g., according to predetermined rules and/or posetemplates) as corresponding to each unique instance of objects. Forexample, the robotic system 100 can identify the groupings of the objectoutlines that correspond to a pattern (e.g., same values or varying at aknown rate/pattern) in color, brightness, depth/location, or acombination thereof across the object lines. Also, for example, therobotic system 100 can identify the groupings of the object outlinesaccording to predetermined shape/pose templates defined in the masterdata.

From the recognized objects in the pickup location, the robotic system100 can select (e.g., according to a predetermined sequence or set ofrules and/or templates of object outlines) one as the target packages112. For example, the robotic system 100 can select the targetpackage(s) 112 as the object(s) located on top, such as according to thepoint cloud representing the distances/positions relative to a knownlocation of the sensor. Also, for example, the robotic system 100 canselect the target package 112 as the object(s) located at a corner/edgeand having two or more surfaces that are exposed/shown in the imagingresults. The available vacuum grippers and/or regions can also be usedto select the target packages. Further, the robotic system 100 canselect the target package 112 according to a predetermined pattern(e.g., left to right, nearest to furthest, etc. relative to a referencelocation).

In some embodiments, the end effector 140 can be configured to gripmultiple instances of the target packages 112 from among the sourcepackage. For example, the robotic system 100 can generate instructionsfor the end effector 140 to engage multiple instances of the vacuumregions 117 to perform the gripping operation to simultaneously gripmultiple instances of the target packages 112. As a specific example,the end effector 140 can be used to execute instructions for thegripping operation of gripping multiple instances of the target package112 separately and in sequence, one after the other. For instance, theinstructions can include performing the gripping operation using one ofthe vacuum regions 117 to grip a first instance of the target package112 that is in one pose or one orientation, then, if necessary,repositioning the end effector 140 to engage a second or differentinstance of the vacuum regions 117 to grip a second instance of thetarget package 112. In another specific example, the end effector 140can be used to execute instructions for the gripping operation ofsimultaneous gripping of separate instances of the target package 112.For instance, the end effector 140 can be positioned to simultaneouslycontact two or more instances of the target package 112 and engage eachof the corresponding instances of vacuum regions 117 to perform thegripping operation on each of the multiple instances of the targetpackage 112. In the above embodiments, each of the vacuum regions 117can be independently operated as necessary to perform the differentgripping operations.

For the selected target package 112, the robotic system 100 can furtherprocess the imaging result to determine the start location 114 and/or aninitial pose. For example, the robotic system 100 can determine theinitial pose of the target package 112 based on selecting from multiplepredetermined pose templates (e.g., different potential arrangements ofthe object outlines according to corresponding orientations of theobject) the one that corresponds to a lowest difference measure whencompared to the grouping of the object outlines. Also, the roboticsystem 100 can determine the start location 114 by translating alocation (e.g., a predetermined reference point for the determined pose)of the target package 112 in the imaging result to a location in thegrid used by the robotic system 100. The robotic system 100 cantranslate the locations according to a predetermined calibration map.

In some embodiments, the robotic system 100 can process the imagingresults of the drop areas to determine open spaces between objects. Therobotic system 100 can determine the open spaces based on mapping theobject lines according to a predetermined calibration map thattranslates image locations to real-world locations and/or coordinatesused by the system. The robotic system 100 can determine the open spacesas the space between the object lines (and thereby object surfaces)belonging to different groupings/objects. In some embodiments, therobotic system 100 can determine the open spaces suitable for the targetpackage 112 based on measuring one or more dimensions of the open spacesand comparing the measured dimensions to one or more dimensions of thetarget package 112 (e.g., as stored in the master data). The roboticsystem 100 can select one of the suitable/open spaces as the tasklocation 116 according to a predetermined pattern (e.g., left to right,nearest to furthest, bottom to top, etc. relative to a referencelocation).

In some embodiments, the robotic system 100 can determine the tasklocation 116 without or in addition to processing the imaging results.For example, the robotic system 100 can place the objects at theplacement area according to a predetermined sequence of actions andlocations without imaging the area. Additionally, the sensors (e.g.,vision sensor device 143) attached to the vacuum gripper assembly 141can output image data used to periodically image the area. The imagingresults can be updated based on the additional image data. Also, forexample, the robotic system 100 can process the imaging result forperforming multiple tasks (e.g., transferring multiple objects, such asfor objects located on a common layer/tier of a stack).

At block 706, the robotic system 100 can calculate a base plan for thetarget package 112. For example, the robotic system 100 can calculatethe base motion plan based on calculating a sequence of commands orsettings, or a combination thereof, for the actuation devices 212 ofFIG. 2 that will operate the robotic system 132 of FIG. 3 and/or theend-effector (e.g., the end-effector 140 of FIGS. 3-5). For some tasks,the robotic system 100 can calculate the sequence and the setting valuesthat will manipulate the robotic system 132 and/or the end-effector 140to transfer the target package 112 from the start location 114 to thetask location 116. The robotic system 100 can implement a motionplanning mechanism (e.g., a process, a function, an equation, analgorithm, a computer-generated/readable model, or a combinationthereof) configured to calculate a path in space according to one ormore constraints, goals, and/or rules. For example, the robotic system100 can use predetermined algorithms and/or other grid-based searches tocalculate the path through space for moving the target package 112 fromthe start location 114 to the task location 116. The motion planningmechanism can use a further process, function, or equation, and/or atranslation table, to convert the path into the sequence of commands orsettings, or combination thereof, for the actuation devices 212. Inusing the motion planning mechanism, the robotic system 100 cancalculate the sequence that will operate the robotic arm 206 (FIG. 3)and/or the end-effector 140 (FIG. 3) and cause the target package 112 tofollow the calculated path. The vision sensor device 143 can be used toidentify any obstructions and recalculate the path and refine the baseplan.

At block 708, the robotic system 100 can begin executing the base plan.The robotic system 100 can begin executing the base motion plan based onoperating the actuation devices 212 according to the sequence ofcommands or settings or combination thereof. The robotic system 100 canexecute a first set of actions in the base motion plan. For example, therobotic system 100 can operate the actuation devices 212 to place theend-effector 140 at a calculated location and/or orientation about thestart location 114 for gripping the target package 112 as illustrated inblock 752.

At block 754, the robotic system 100 can analyze the position of objectsusing sensor information (e.g., information from the vision sensordevice 143, sensors 216, force detector assembly 205) obtained beforeand/or during the gripping operation, such as the weight of the targetpackage 112, the center of mass of the target package 112, the relativeposition of the target package 112 with respect to vacuum regions, or acombination thereof. The robotic system 100 can operate the actuationdevices 212 and vacuum source 221 (FIG. 7) to have the end-effector 140engage and grip the target package 112. The image data from the visionsensor device 143 and/or data from the force sensor assembly 205 can beused to analyze the position and number of the target packages 112. Atblock 755, the vision sensor device 143 can be used to verify theposition of the end effector 140 relative to target packages 112 orother objects. In some embodiments, as illustrated at block 756, therobotic system 100 can perform an initial lift by moving theend-effector up by a predetermined distance. In some embodiments, therobotic system 100 can reset or initialize an iteration counter ‘i’ usedto track a number of gripping actions.

At block 710, the robotic system 100 can measure the established grip.The robotic system 100 can measure the established grip based onreadings from the force detector assembly 205 of FIG. 7, vision sensordevice 143, or other sensors, such as the pressure sensors 434 (FIG.15). For example, the robotic system 100 can determine the gripcharacteristics by using one or more of force detector assembly 205 ofFIG. 3 to measure a force, a torque, a pressure, or a combinationthereof at one or more locations on the robotic arm 139, one or morelocations on the end-effector 140, or a combination thereof. In someembodiments, such as for the grip established by the assembly 141,contact or force measurements can correspond to a quantity, a location,or a combination thereof of the suction elements (e.g., suction elements416 of FIG. 14) contacting a surface of the target package 112 andholding a vacuum condition therein. Additionally or alternative, thegrip characteristic can be determined based on output from the visionsensor device 143. For example, image data from the sensor detector 143can be used to determine whether the object moves relative to the endeffector 140 during transport.

At decision block 712, the robotic system 100 can compare the measuredgrip to a threshold (e.g., an initial grip threshold). For example, therobotic system 100 can compare the contact or force measurement to apredetermined threshold. The robot system 100 can also compare imagedata from the detector 143 to reference image data (e.g., image datacaptured at initial object pickup) to determine whether the grippedobjects have moved, for example, relative to one another or relative tothe gripper assembly 141. Accordingly, the robotic system 100 candetermine whether the contact/grip is sufficient to continuemanipulating (e.g., lifting, transferring, and/or reorienting) thetarget package(s) 112.

When the measured grip fails to satisfy the threshold, the roboticsystem 100 can evaluate whether the iteration count for regripping thetarget packages(s) 112 has reached an iteration threshold, asillustrated at decision block 714. While the iteration count is lessthan the iteration threshold, the robotic system 100 can deviate fromthe base motion plan when the contact or force measurement fails tosatisfy (e.g., is below) the threshold. Accordingly, at block 720, therobotic system 100 can operate the robotic arm 139 and/or theend-effector 140 to execute a regripping action not included in the basemotion plan. For example, the regripping action can include apredetermined sequence of commands or settings, or a combinationthereof, for the actuation devices 212 that will cause the robotic arm139 to lower the end-effector 140 (e.g., in reversing the initial lift)and/or cause the end-effector 140 to release the target package(s) 112and regrip the target package(s) 112. In some embodiments, thepredetermined sequence can further operate the robotic arm 139 to adjusta position of the gripper after releasing the target object and beforeregripping it or altering the areas at which the vacuum is drawn. Inperforming the regripping action, the robotic system 100 can pauseexecution of the base motion plan. After executing the regrippingaction, the robotic system 100 can increment the iteration count.

After regripping the object, the robotic system 100 can measure theestablished grip as described above for block 710 and evaluate theestablished grip as described above for block 712. The robotic system100 can attempt to regrip the target package 112 as described aboveuntil the iteration count reaches the iteration threshold. When theiteration count reaches the iteration threshold, the robotic system 100can stop executing the base motion plan, as illustrated at block 716. Insome embodiments, the robotic system 100 can solicit operator input, asillustrated at block 718. For example, the robotic system 100 cangenerate an operator notifier (e.g., a predetermined message) via thecommunication devices 206 of FIG. 2 and/or the input-output devices 208of FIG. 2. In some embodiments, the robotic system 100 can cancel ordelete the base motion plan, record a predetermined status (e.g., anerror code) for the corresponding task, or perform a combinationthereof. In some embodiments, the robotic system 100 can reinitiate theprocess by imaging the pickup/task areas (block 702) and/or identifyinganother item in the pickup area as the target object (block 704) asdescribed above.

When the measured grip (e.g., measured grips for each retained package)satisfies the threshold, the robotic system 100 can continue executingremaining portions/actions of the base motion plan, as illustrated atblock 722. Similarly, when the contact measure satisfies the thresholdafter regripping the target package 112, the robotic system 100 canresume execution of the paused base motion plan. Accordingly, therobotic system 100 can continue executing the sequenced actions (i.e.,following the grip and/or the initial lift) in the base motion plan byoperating the actuation devices 212 and/or the transport motor 214 ofFIG. 2 according to the remaining sequence of commands and/or settings.For example, the robotic system 100 can transfer (e.g., verticallyand/or horizontally) and/or reorient the target package 112 according tothe base motion plan.

While executing the base motion plan, the robotic system 100 can trackthe current location and/or the current orientation of the targetpackage 112. The robotic system 100 can track the current locationaccording to outputs from the position sensors 224 of FIG. 2 to locateone or more portions of the robotic arm and/or the end-effector. In someembodiments, the robotic system 100 can track the current location byprocessing the outputs of the position sensors 224 with acomputer-generated model, a process, an equation, a position map, or acombination thereof. Accordingly, the robotic system 100 can combine thepositions or orientations of the joints and the structural members andfurther map the positions to the grid to calculate and track the currentlocation 424. In some embodiments, the robotic system 100 can includemultiple beacon sources. The robotic system 100 can measure the beaconsignals at one or more locations in the robotic arm and/or theend-effector and calculate separation distances between the signalsources and the measured location using the measurements (e.g., signalstrength, time stamp or propagation delay, and/or phase shift). Therobotic system 100 can map the separation distances to known locationsof the signal sources and calculate the current location of thesignal-receiving location as the location where the mapped separationdistances overlap.

At decision block 724, the robotic system 100 can determine whether thebase plan has been fully executed to the end. For example, the roboticsystem 100 can determine whether all of the actions (e.g., the commandsand/or the settings) in the base motion plan 422 have been completed.Also, the robotic system 100 can determine that the base motion plan isfinished when the current location matches the task location 116. Whenthe robotic system 100 has finished executing the base plan, the roboticsystem 100 can reinitiate the process by imaging the pickup/task areas(block 702) and/or identifying another item in the pickup area as thetarget object (block 704) as described above.

Otherwise, at block 726, the robotic system 100 can measure the grip(i.e., by determining the contact/force measurements) during transfer ofthe target package 112. In other words, the robotic system 100 candetermine the contact/force measurements while executing the base motionplan. In some embodiments, the robotic system 100 can determine thecontact/force measurements according to a sampling frequency or atpredetermined times. In some embodiments, the robotic system 100 candetermine the contact/force measurements before and/or after executing apredetermined number of commands or settings with the actuation devices212. For example, the robotic system 100 can sample the contact sensors226 after or during a specific category of maneuvers, such as for liftsor rotations. Also, for example, the robotic system 100 can sample thecontact sensors 226 when a direction and/or a magnitude of anaccelerometer output matches or exceeds a predetermined threshold thatrepresents a sudden or fast movement. The robotic system 100 candetermine the contact/force measurements using one or more processesdescribed above (e.g., for block 710).

In some embodiments, the robotic system 100 can determine theorientation of the gripper and/or the target package 112 and adjust thecontact measure accordingly. The robotic system 100 can adjust thecontact measure based on the orientation to account for a directionalrelationship between a sensing direction for the contact sensor andgravitational force applied to the target object according to theorientation. For example, the robotic system 100 can calculate an anglebetween the sensing direction and a reference direction (e.g., “down” orthe direction of the gravitational force) according to the orientation.The robotic system 100 can scale or multiply the contact/forcemeasurement according to a factor and/or a sign that corresponds to thecalculated angle.

At decision block 728, the robotic system 100 can compare the measuredgrip to a threshold (e.g., a transfer grip threshold). In someembodiments, the transfer grip threshold can be less than or equal tothe initial grip threshold associated with evaluating an initial (e.g.,before transferring) grip on the target package 112. Accordingly, therobotic system 100 can enforce a stricter rule for evaluating the gripbefore initiating transfer of the target package 112. The thresholdrequirement for the grip can be higher initially since contactsufficient for picking up the target package 112 is likely to besufficient for transferring the target package 112.

When the measured grip satisfies (e.g., is not less than) the thresholdand the correct packages are gripped (e.g., determined based on theimage data from the vision sensor device 143), the robotic system 100can continue executing the base plan as illustrated at block 722 anddescribed above. When the measured grip fails to satisfy (e.g., is lessthan) the threshold or the correct packages are not gripped, the roboticsystem 100 can deviate from the base motion plan and execute one or moreresponsive actions as illustrated at block 730. When the measured gripis insufficient in light of the threshold, the robotic system 100 canoperate the robotic arm 139, the end-effector, or a combination thereofaccording to commands and/or settings not included in the base motionplan. In some embodiments, the robotic system 100 can execute differentcommands and/or settings based on the current location.

For illustrative purposes, the response actions will be described usinga controlled drop. However, it is understood that the robotic system 100can execute other actions, such as by stopping execution of the basemotion plan as illustrated at block 716 and/or by soliciting operatorinput as illustrated at block 718.

The controlled drop includes one or more actions for placing the targetpackage 112 in one of the drop areas (e.g., instead of the task location116) in a controlled manner (i.e., based on lowering and/or releasingthe target package 112 and not as a result of a complete grip failure).In executing the controlled drop, the robotic system 100 can dynamically(i.e., in real time and/or while executing the base motion plan)calculate different locations, maneuvers or paths, and/or actuationdevice commands or settings according to the current location. In someembodiments, end effector 140 can be configured for a grip releaseoperation for multiple instances of the target package 112. For example,in some embodiments, the end effector 140 can be configured forsimultaneously or sequentially performing the grip release operation byselectively disengage the vacuum regions 117 as necessary to releaseeach instance of the target package 112 accordingly. The robotic system100 can select whether to simultaneously or sequentially release objectsand the order of release based on the position of the retained objects,object arrangement at the drop area, etc.

At block 762, the robotic system 100 can calculate the adjusted droplocation and/or an associated pose for placing the target package 112.In calculating the adjusted drop location, the robotic system 100 canidentify the drop area (e.g., the source drop area, the destination droparea, or the transit drop area) nearest to and/or ahead (e.g., betweenthe current location and the task location) of the current location.Also, when the current location is between (i.e., not within) the dropareas, the robotic system 100 can calculate distances to the drop areas(e.g., distances to representative reference locations for the dropareas). Accordingly, the robotic system 100 can identify the drop areathat is nearest to the current location and/or ahead of the currentlocation. Based on the identified drop area, the robotic system 100 cancalculate a location therein as the adjusted drop location. In someembodiments, the robotic system 100 can calculate the adjusted droplocation based on selecting a location according to a predeterminedorder (e.g., left to right, bottom to top, and/or front to back relativeto a reference location).

In some embodiments, the robotic system 100 can calculate distances fromthe current location to open spaces (e.g., as identified in block 704and/or tracked according to ongoing placements of objects) within thedrop areas. The robotic system 100 can select the open space that isahead of the current location and/or nearest to the current location 424as the adjusted drop location.

In some embodiments, prior to selecting the drop area and/or the openspace, the robotic system 100 can use a predetermined process and/orequation to translate the contact/force measure to a maximum transferdistance. For example, the predetermined process and/or equation canestimate based on various values of the contact measure a correspondingmaximum transfer distance and/or a duration before a complete gripfailure. Accordingly, the robotic system 100 can filter out theavailable drop areas and/or the open spaces that are farther than themaximum transfer distance from the current location. In someembodiments, when the robotic system 100 fails to identify availabledrop areas and/or open spaces (e.g., when the accessible drop areas arefull), the robotic system 100 can stop executing the base motion plan,as illustrated at block 716, and/or solicit operator input, asillustrated at block 718.

At block 766, the robotic system 100 can calculate the adjusted motionplan for transferring the target package 112 from the current locationto the adjusted drop location. The robotic system 100 can calculate theadjusted motion plan in a way similar to that described above for block506.

At block 768, the robotic system 100 can execute the adjusted motionplan in addition to and/or instead of the base motion plan. For example,the robotic system 100 can operate the actuation devices 212 accordingto the sequence of commands or settings or combination thereof, therebymaneuvering the robotic arm 139 and/or the end-effector to cause thetarget package 112 to move according to the path.

In some embodiments, the robotic system 100 can pause execution of thebase motion plan and execute the adjusted motion plan. Once the targetpackage 112 is placed at the adjusted drop location based on executingthe adjusted motion plan (i.e., completing execution of the controlleddrop), in some embodiments, the robotic system 100 can attempt to regripthe target package 112 as described above for block 720 and then measurethe established grip as described above for block 710. In someembodiments, the robotic system 100 can attempt to regrip the targetpackage 112 up to an iteration limit as described above. If the contactmeasure satisfies the initial grip threshold, the robotic system 100 canreverse the adjusted motion plan (e.g., return to the pausedpoint/location) and continue executing the remaining portions of thepaused base motion plan. In some embodiments, the robotic system 100 canupdate and recalculate the adjusted motion plan from the currentlocation 424 (after regripping) to the task location 116 and execute theadjusted motion plan to finish executing the task.

In some embodiments, the robotic system 100 can update an area log(e.g., a record of open spaces and/or placed objects) for the accesseddrop area to reflect the placed target package 112. For example, therobotic system 100 can regenerate the imaging results for thecorresponding drop area. In some embodiments, the robotic system 100 cancancel the remaining actions of the base motion plan after executing thecontrolled drop and placing the target package 112 at the adjusted droplocation. In one or more embodiments, the transit drop area can includea pallet or a bin placed on top of one of the transport units 106 ofFIG. 1. At a designated time (e.g., when the pallet/bin is full and/orwhen the incoming pallet/bin is delayed), the corresponding transportunit can go from the drop area to the pickup area. Accordingly, therobotic system 100 can reimplement the method 500, thereby reidentifyingthe dropped items as the target package 112 and transferring them to thecorresponding task location 116.

Once the target package 112 has been placed at the adjusted droplocation, the robotic system 100 can repeat the method 700 for a newtarget object. For example, the robotic system 100 can determine thenext object in the pickup area as the target package 112, calculate anew base motion plan to transfer the new target object, etc.

In some embodiments, the robotic system 100 can include a feedbackmechanism that updates the path calculating mechanism based on thecontact measure 312. For example, as the robotic system 100 implementsthe actions to regrip the target package 112 with adjusted positions(e.g., as described above for block 720), the robotic system 100 canstore the position of the end-effector that produced the contact/forcemeasurements that satisfied the threshold (e.g., as described above forblock 712). The robotic system 100 can store the position in associationwith the target package 112. The robotic system 100 can analyze thestored positions (e.g., using a running window for analyzing a recentset of actions) for gripping the target package 112 when the number ofgrip failures and/or successful regrip actions reach a threshold. When apredetermined number of regrip actions occur for a specific object, therobotic system 100 can update the motion planning mechanism to place thegripper at a new position (e.g., position corresponding to the highestnumber of successes) relative to the target package 112.

Based on the operations represented in block 710 and/or block 726 therobotic system 100 (via, e.g., the processors 202) can track a progressof executing the base motion plan. In some embodiments, the roboticsystem 100 can track the progress according to horizontal transfer ofthe target package(s) 112. The robotic system 100 can track the progressbased on measuring the established grip (block 710) before initiatingthe horizontal transfer and based on measuring the grip during transfer(block 726) after initiating the horizontal transfer. Accordingly, therobotic system 100 can selectively generate a new set (i.e., differentfrom the base motion plan) of actuator commands, actuator settings, or acombination thereof based on the progress as described above.

In other embodiments, for example, the robotic system 100 can track theprogress based on tracking the commands, the settings, or a combinationthereof that has been communicated to and/or implemented by theactuation devices 212. Based on the progress, the robotic system 100 canselectively generate the new set of actuator commands, actuatorsettings, or a combination thereof to execute the regrip response actionand/or the controlled drop response action. For example, when theprogress is before any horizontal transfer of the target package 112,the robotic system 100 can select the initial grip threshold and executethe operations represented in blocks 712 (via, e.g., function calls orjump instructions) and onward. Also, when the progress is after thehorizontal transfer of the target package 112, the robotic system 100can select the transfer grip threshold and execute the operationsrepresented in blocks 728 (via, e.g., function calls or jumpinstructions) and onward.

Implementing granular control/manipulation of the target package 112(i.e., choosing to implement the base motion plan or deviate from it)according to the contact/force measurement and vision-based monitoring,via the imaging data from the vision sensor device 143, providesimproved efficiency, speed, and accuracy for transferring the objects.For example, regripping the target packages 112 when the contact measureis below the initial grip threshold or the packages 112 are improperlypositioned decreases the likelihood of grip failure occurring duringtransfer, which decreases the number of objects lost or unintentionallydropped during transfer. The vacuum regions and vacuum levels can beadjusted to maintain the desired grip and to further enhance handling ofthe packages 112. Moreover, each lost object requires human interactionto correct the outcome (e.g., move the lost object out of the motionpath for subsequent tasks, inspect the lost object for damages, and/orcomplete the task for the lost object). Thus, reducing the number oflost objects reduces the human effort necessary to implement the tasksand/or the overall operation.

FIGS. 18-21 illustrate stages of robotically gripping and transportingobjects according to the method 490 of FIG. 16 or method 700 of FIG. 17in accordance with one or more embodiments of the present disclosure.FIG. 18 shows the gripper assembly 141 located above a stack of objects.The robotic arm 139 can positioned the gripper assembly 141 directlyabove targeted objects. A controller can analyze image data from thevision sensor device 143 to identify, for example, the target objects812 a, 812 b, as discussed at block 704 of FIG. 17. A plan (e.g., pickupor base plan) can be generated based on collected image data. The plancan be generated based on (a) a carrying capability of the gripperassembly 141 and/or (b) a configuration of target objects.

FIG. 19 shows the lower surface of the gripper assembly 141 overlayingthe target objects 812 a, 812 b and a large non-targeted object 818.Output from the vision sensor device 143 can be analyzed to confirm theposition of the gripper assembly 141 relative to the targeted objects.Based on the position of the objects 812 a, 812 b, the vacuum regions117 a, 117 b are identified for drawing a vacuum. In some embodiments,readings from the force sensor 203 are used to confirm the gripperassembly 141 has contacted the upper surfaces of a stack 814 prior toand/or after gripping target objects 812 a, 812 b.

FIG. 20 shows air being sucked into the vacuum regions 117 a, 117 b, asindicated by arrows, to hold the target objects 812 a, 812 b against thegripper assembly 141 without drawing a vacuum (or a substantial vacuum)at the other vacuum region 117 c. The vacuum level can be increased ordecreased to increase or decrease the compression of the compliantpanel(s) 412 (one identified). The vacuum grip can be evaluated asdiscussed in connection with block 710 of FIG. 17.

FIG. 21 shows the raised gripper assembly 141 securely holding thetarget objects 812 a, 812 b. The vision sensor device 143 can be used tomonitor the positions of the target objects 812 a, 812 b. Additionallyor alternatively, the force detector assembly 205 can be used todetermine information about the load, such as the positions and weightof the target objects 812 a, 812 b. The vacuum regions 117 a, 117 b cancontinue to suck in air to securely hold the targeted objects 812 a, 812b. The vacuum grip can be monitored during transfer, as discussed atblock 726 of FIG. 17. The applied vacuum can be stopped or reduced torelease the objects 812 a, 812 b. This process can be repeated totransfer each of the objects in the stack.

Grasp Set

FIG. 22 illustrates example aspects of a grasp set 2200 in accordancewith one or more embodiments of the present technology. The grasp set2200 can include one or more grip poses (e.g., positions, locations,orientations, etc.) of the end effector 140 relative to the targetpackage 112. More specifically, the grip poses can represent therelative position between the target package 112 and the end effector140 when determining whether the grip poses can be used for gripping thetarget package 112.

As an illustrative example of the grip poses, FIG. 22 illustrates afirst pose 2202 and a second pose 2204 for the end effector 140 relativeto the target package 112. The first pose 2202 is illustrated via afirst top-view 2212 and a first side-view 2214 representative of thefirst pose 2202 of the end effector 140 for grasping the target package112. The target package 112 may have a smaller footprint than the endeffector 140 and be obscured by the end effector 140 in the firsttop-view 2212. Accordingly, for the first top-view 2212, the targetpackage 112 is shown via dashed lines to indicate the pose relative tothe end effector 140. Similar to the first pose 2202, the second pose2204 is illustrated via a second top-view 2222 and a second side-view2224 representative of a second pose of the end effector 140 forgrasping the target package 112.

The first pose 2202 can have lengths of the end effector 140 and thetarget package 112 parallel to each other. The first pose 2202 and thesecond pose 2204 can be rotated/offset by 90 degrees about a verticaldirection (e.g., z-axis, not shown in FIG. 22). Accordingly, the secondpose 2204 can have the length of the end effector 140 parallel with thewidth of the target package 112 and orthogonal to the length of thetarget package 112.

The robotic system 100 of FIG. 1 can derive the grasp set 2200 byidentifying a grip pose for the target package 112 and overlaying orarranging a model of the end effector 140 at one or more grip posesrelative to a model of the target package 112 at the identified grippose. The robotic system 100 can follow a predetermined pattern orroutine in arranging and analyzing the models. In some embodiments, thegrasp set 2200 can include notified grip poses 2206 of the end effector140 with one or more edges/boundaries thereof aligned with one or morecorresponding peripheral edges of the target package 112. For example,the notified grip poses 2206 can have one or more peripheral edges ofthe gripping interface 121 of FIG. 4 and/or the vacuum regions 117 ofFIG. 4 can be aligned with corresponding peripheral edges of the targetpackage 112. In other words, the peripheral edge of the target package112 and the corresponding peripheral edge of the vacuum region 117 canbe coincident with a vertical line/plane. In one or more embodiments,the grasp set 2200 can be a notified grasp set that includes thenotified grip poses 2206 without non-aligned end effector grip poses.The robotic system 100 can derive the grasp set 2200 offline (e.g.,before receiving and/or processing actual packages) via computer modelsof expected or known packages. Alternatively or additionally, therobotic system 100 can derive the grasp set 2200 dynamically, such asbased on a real-time image depicting an actual package targeted fortransfer.

The grasp set 2200 can further include movement control parameters 2208associated with the grip poses. The movement control parameters 2208 caneach include an indication identifying the vacuum regions 117 and/or thesuction elements 151 required to grasp the target object for thecorresponding grip pose. Also, the movement control parameters 2208 caninclude a value that represents a speed, an acceleration, a force, arate, or the like used to control movement of the end effector 140 whilegrasping and transferring the target package 112. For example, themovement control parameters 2208 can include a transfer speed multiplier(TSM) for each grip pose (e.g., a first TSM 2218 for the first pose 2202and a second TSM 2228 for the second pose 2040. The TSM can include avalue in the range R∈[0,1], wherein ‘1’ represents full or maximum speedand ‘0’ represents stop or no movement.

The robotic system 100 can derive or calculate (offline and/ordynamically) the movement control parameters 2208 according to thecorresponding grip poses. The robotic system 100 can derive the movementcontrol parameters 2208 based on an overlap between the end effector 140and the target package 112, information regarding physical aspects ofthe target package 112, and/or records from previous transfers of thesame type of package. For example, the robotic system 100 can identifythe overlapped area and the corresponding vacuum regions 117 and/orsuction elements 151. Further, the robotic system 100 can derive themovement control parameters 2208 using a predetermined function thattakes as input a size of the overlapped area and/or the number ofsuction elements 151 over the target package 112. The predeterminedfunction may further use a weight, a CoM information, one or moredimensions, a surface type, and/or other information in the master dataregarding the target package 112. In some embodiments, the roboticsystem 100 can automatically adjust the movement control parameters 2208(e.g., the TSM) based on occurrences (e.g., for decreasing theparameters) or prolonged absence (e.g., for increasing the parameters)of initial grip failures and/or package loss during transfer.

When transferring a package, the robotic system 100 can select a grippose from the grasp set 2200 having the maximum instance of the movementcontrol parameters 2208 (e.g., a highest value of the TSMs).Accordingly, the robotic system 100 can reduce the transfer duration forthe corresponding package. Additionally, the robotic system 100 canconsider and analyze simultaneously grasping and transferring multiplepackages. The robotic system 100 can analyze the feasibility of thesimultaneous transfer, and when feasible, the robotic system 100 caneffectively compare the efficiency (e.g., total transfer times) of thesimultaneous transfer to that of separate individual transfers.

As an illustrated example, the first TSM 2218 may be greater than thesecond TSM 2228 since the first pose 2202 provides greater overlapbetween the end effector 140 and the target package 112 in comparison tothe second pose 2204. In other words, the first TSM 2218 can be ‘X’ andthe second TSM 2228 can be ‘Y’, where ‘X>Y.’ Accordingly, in the absenceof simultaneous grasp availability or consideration, the robotic system100 can select the first pose 2202 over the second pose 2204.

For the example illustrated in FIG. 22, the robotic system 100 candetermine that the end effector 140 extends over a simultaneous grasptarget 2250. The simultaneous grasp target 2250 can be a package that isadjacent to the target package 112 and/or located in the same layer asthe target package 112. In some embodiments, the robotic system 100 canvalidate the adjacent package as the simultaneous grasp target 2250 forfurther processing when the adjacent package has a top portion orsurface that is at the same height as or within a threshold range from atop surface/portion height of the target package 112. The robotic system100 can analyze feasibility of simultaneously grasping and transferringthe target package 112 with the simultaneous grasp target 2250, such asby deriving or determining the overall movement control parameter and/orby analyzing a release sequence. When the analysis result indicatesfeasibility of simultaneous grasp/transfer, the robotic system 100 canderive the corresponding motion plan and/or evaluate the correspondingefficiencies. Accordingly, the robotic system 100 can reduce the overalltransfer time for the layer/stack of packages by grasping andtransferring multiple packages when applicable and beneficial. Detailsregarding the simultaneous grasp and transfer are described below.

Simultaneous Grasp of Multiple Packages

FIGS. 23A-23F illustrate example scenarios for simultaneouslytransferring multiple objects in accordance with one or more embodimentsof the present technology. The illustrated example scenarios showoperating states of the vacuum regions 117 of the end effector 140 andthe corresponding effects on the target package 112 and the simultaneousgrasp target 2250. Accordingly, the illustrated example scenarios showerror conditions and corresponding solutions associated with thesimultaneous grasp/transfer.

For the illustrative example, the target package 112 can have adimension that is longer than a corresponding dimension of the firstvacuum region 117 a. For the notified grip pose illustrated in FIG. 23A,the target package 112 can extend into the second vacuum region 117 b.The simultaneous grasp target 2250 may be adjacent to the target package112. The two packages may be separated by a distance. With the selectedgrip pose, the second vacuum region 117 b and the third vacuum region117 c can both overlap the simultaneous grasp target 2250. Accordingly,the robotic system 100 can activate all three vacuum regions tosimultaneously grasp and pick the simultaneous grasp target 2250 and thetarget package 112.

FIGS. 23B and 23C illustrate possible error conditions for releasing thepackages. As illustrated in FIG. 23B, deactivating only the first vacuumregion 117 a may not fully release the target object 112. For example,packages with relatively lighter weight, flexible structure, and/orsmoother grasp surface may remain adhered to or grasped by the secondvacuum region 117 b due to the overlap. As a result, the target object112 may collide with another object, remain grasped by the end effector140, and/or fall in an unexpected manner. Alternatively, as illustratedin FIG. 23C, deactivating both the first vacuum region 117 a and thesecond vacuum region 117 b to release the target object 112 may causeunexpected release of the simultaneous grasp target 2250. For example,one vacuum region may not be sufficient for grasp the simultaneous grasptarget 2250 when it is heavier, is rigid, and/or has an irregular orporous grasp surface.

FIGS. 23D-23F illustrate potential solutions for processing the targetpackage 112 and the simultaneous grasp target 2250. As illustrated inFIG. 23D, the robotic system 100 can analyze alternative releasesequences. The robotic system 100 can consider releasing thesimultaneous grasp target 2250 before the target package 112.Accordingly, the robotic system 100 can consider deactivating the secondvacuum region 117 b and the third vacuum region 117 c to release thesimultaneous grasp target 2250. The robotic system 100 can analyze theeffects of such deactivation on the target package 112. For the exampleconditions discussed above, the first vacuum region 117 a can besufficient to grasp the target package 112, and the alternative releasesequence can be a viable solution. In some embodiments, the roboticsystem 100 can process the alternative release sequences by rearrangingthe target and simultaneously grasped designations for the same set ofpackages (e.g., by designating Target 2 in FIG. 23D as the targetpackage 112 and Target 1 as the simultaneously grasped package 2250)while maintaining that the target package 112 is released before thesimultaneously grasped package.

Alternatively or additionally, the robotic system 100 can consider othergrip poses as illustrated in FIG. 23E. For example, the initiallyanalyzed grip pose of the end effector 140 can have an outer edge of thefirst vacuum region 117 a aligned with a first peripheral edge of thetarget package 112. The robotic system 100 can additionally oralternatively process and analyze an alternate pose 2402 that aligns aninner boundary of the first vacuum region 117 a with a second peripheraledge of the target package 112 that is opposite the first peripheraledge. Effectively, the robotic system 100 can consider shifting the endeffector 140 along a lateral direction relative to the target package112 and/or the simultaneously grasped package 2250. For the exampleconditions discussed above, the target package 112 can be overlapped andgrasped by the first vacuum region 117 a without being overlapped by thesecond vacuum region 117 b. Accordingly, the second vacuum region 117 bcan be dedicated to grasping only the simultaneously grasped package2250. Thus, the target package 112 can be released by deactivating thefirst vacuum region 117 a, and independently, the simultaneously graspedpackage 2250 can be released by deactivating the second vacuum region117 b and the third vacuum region 117 c.

FIG. 23F illustrates the robotic system 100 grasping only the targetpackage 112. For example, the robotic system 100 can revert to graspingand transferring the target package 112 without simultaneously graspingother packages when the analysis for the simultaneous grasp indicatesviolations of any rules or thresholds.

Contact Evaluation

FIG. 24 illustrates example gripper placement conditions in accordancewith one or more embodiments of the present technology. In someembodiments, the robotic system 100 can derive an overlap measurerepresenting an amount of overlap between a package and a correspondingvacuum region. For the example illustrated in FIG. 24, the end effector140 can be positioned such that the second vacuum region 117 b fullyoverlaps the target package 112. Accordingly, the robotic system 100 candetermine the overlap measure for the second vacuum region 117 b as CALLWhen the vacuum region does not overlap the target package 112 (notshown in FIG. 24), the robotic system 100 can determine the overlapmeasure as ‘NONE’.

The robotic system 100 can have one or more values to describe partialoverlaps between the vacuum regions and the target package 112. In someembodiments, the robotic system 100 can determine the partial overlapmeasure between one or more of the vacuum regions (i.e. the first vacuumregion 117 a, the second vacuum region 117 b, or the third vacuum region117 c) and the target object 112 according to sensor readings, such asfrom a vacuum sensor (i.e. the sensor 216). For example, if the vacuumsensors are unable to detect that the target object 112 is in contactwith one of the vacuum regions (i.e. the first vacuum region 117 a) bychange in the vacuum pressure even though the grasp pose for grippingindicates overlap of or contact between the target object 112 and thevacuum region, the robotic system 100 can classify the overlap as“TOUCH” as illustrated in FIG. 24. In some embodiments, the roboticsystem 100 can determine the partial overlap measures according tocorresponding sets or ranges of thresholds. For the example illustratedin FIG. 24, the robotic system 100 can use 50% overlap as a dividingthreshold between a ‘TOUCH’ classification and a ‘SOME’ classification.Accordingly, the robotic system 100 can determine the overlap measurefor the first vacuum region 117 a as ‘TOUCH’ since less than 50% of theregion overlaps the target package 112. Further, the robotic system 100can determine the overlap measure for the third vacuum region 117 c as‘SOME’ since the overlap amount corresponds to a value greater than 50%and less than 100%. In some embodiments, the distinction between the‘SOME’ classification and the ‘ALL’ classification can be based on athreshold value between 50% and 100% overlap.

Real-Time Verification of Grasped Packages

FIG. 25 illustrates an example task location in accordance with one ormore embodiments of the present technology. In some embodiments, a setof destination sensors 2502 may be configured to track a progress of thepackage transfer. For example, the destination sensors 2502 can includeline sensors (e.g., optical sensors) that transmit optical signalsand/or detect changes in the optical signals caused by transferredpackages and/or robotic units (e.g., the end effector 140). Some exampleline sensors can detect absence of corresponding laser or opticalsignals to indicate crossing or entry events and subsequent detectionsof the laser/optical signals to indicate exit events.

In some embodiments, the destination sensors 2502 can be located abovethe task location 116 of FIG. 1 (e.g., the conveyor 120 of FIG. 3). Thedestination sensors 2502 can include a set of deceleration sensors 2504and/or a set of release sensors 2506. The deceleration sensors 2504 caninclude the line sensors configured to trigger deceleration in descentof the transferred package(s) in preparation for release thereof. Therelease sensors 2506 can include the line sensors configured to triggerrelease (via, e.g., deactivation of corresponding vacuum regions 117) ofthe grasped package for placing the package onto the correspondingtarget location 116.

The destination sensors 2502 can be arranged and oriented along one ormore lateral planes. In some embodiments, for example, the destinationsensors 2502 can be arranged along a lateral line (e.g. along thex-direction) and/or according to a fixed separation distance. Thedestination sensors 2502 can be configured to detect crossings along anorthogonal lateral line (e.g., along the y-direction). In other words,the destination sensors 2502 can be configured to detectchanges/disruptions in optical signals that travel along they-direction. Also, the deceleration sensors 2504 can correspond to alateral plane (e.g., a horizontal plane) located above another lateralplane (e.g., a second horizontal plane) that corresponds to the releasesensors 2506.

The robotic system 100 can use the destination sensors 2502 to determineor verify other physical aspects of the transferred packages. Forexample, the robotic system 100 can use the crossing event to determineheights of the transferred packages. The detection lines/planes of thedestination sensors 2502 can be at known heights. Accordingly, therobotic system 100 can determine the package heights by identifying theheight of the end effector 140 at the time of crossing event andcalculating a difference between the identified height and the knownheights of the destination sensors 2502. Also, the robotic system 100can identify the triggered instances of the linearly arranged sensors todetermine a corresponding lateral dimension of the transferred package.As illustrated in FIG. 25, the robotic system 100 can determine thatsensors D1 a, D1 b, and D2 a have detected a crossing event whilesensors D2 b and onward remain undisturbed. Accordingly, the roboticsystem 100 can estimate a width or a length for the simultaneous grasptarget 2250.

The robotic system 100 can use the derived information to verify thetransferred package and the remaining portions of the correspondingmotion plan. For example, the robotic system 100 can further derive andimplement the motion plan according to a rule to release the tallestpackage first. Accordingly, the robotic system 100 can verify that thepackage intended to be released first crosses the sensing line/planebefore other simultaneously transferred package(s). Further, the roboticsystem 100 can compare the sensor-based height and/or lateral dimensionwith known or expected dimensions of the transferred package to verifythe identity/category thereof.

For the example illustrated in FIG. 25, the robotic system 100 canderive and implement a motion plan to simultaneously grasp and transferthe target package 112 and the simultaneous grasp target 2250. Therobotic system 100 can derive the motion plan according to data andanalysis results that indicate the target package 112 being taller thanthe simultaneous grasp target 2250. Thus, the motion plan can correspondto releasing the target package 112 before the simultaneous grasp target2250. However, the utilized dimensions may be erroneous and the actualdimensions of the package at the corresponding location may bedifferent. Accordingly, the robotic system 100 can detect an errorcondition when sensors D1 a-D2 b (e.g., sensors corresponding to thesimultaneous grasp target 2250) indicate crossing events before sensorD3 a and D3 b (e.g., sensors corresponding to the target package 112)indicate crossing events. Additionally or alternatively, the roboticsystem 100 can derive an estimated lateral dimension and/or anunexpected dimension status based on triggering of sensors D1 a-D2 b. Inother words, based on the locations of the triggered sensors, therobotic system 100 can determine that a lateral dimension of thetransferred object is not as expected. As discussed in detail below, therobotic system 100 can respond to the detected error conditions andevaluate the remaining portion of the motion plan. Based on theevaluation, the robotic system 100 can continue with the remainingportions or update/replace the remaining portions.

Operational Flow for Simultaneously Processing Multiple Objects

FIG. 26 is another flow diagram of an example method 2600 for operatinga robotic system (e.g., the robotic system 100) in accordance with oneor more embodiments of the present technology. The method 2600 can befor evaluating simultaneous grasp of two or more objects using amulti-gripper assembly (e.g., the end effector 140 of FIG. 4). Themethod 2600 can be for determining whether simultaneously grasping andtransferring the two or more objects is feasible and/or optimal. Basedon the determination, the robotic system 100 (via, e.g., the processors202 of FIG. 2) can implement the method 2600 to derive and implement oneor more motions plans for grasping and transferring the packages. Themethod 2600 can be implemented based on executing the instructionsstored on one or more of the storage devices 204 of FIG. 2 with one ormore of the processors 202 of FIG. 2. In implementing the motion planand/or the method 2600, the processors 202 can send the motion plan oran associated set/sequence of commands/settings to the transfer assembly104 of FIG. 3 and/or the end effector 140 of FIG. 3. Accordingly, thetransfer assembly 104 and/or the end effector 140 can execute the motionplan to grasp and transfer the packages.

At block 500, the robotic system 100 (e.g., the computer 109 of FIG. 1and/or the processors 202) can receive the image data representative ofat least a portion of an environment as described above. For example,the robotic system 100 can receive from the imaging system 160 of FIG. 3the image data representative of at least a portion of the stack 165FIG. 3 at the pickup environment 163 FIG. 3. The pickup environment 163can include the target package 112 of FIG. 1 and the simultaneous grasptarget 2250 of FIG. 22. Accordingly, the image data can depict thetarget package 112, the simultaneous grasp target 2250, and/or otherpackages in the stack 165 (e.g., the packages forming the top layer ofthe stack 165).

At block 502, the robotic system 100 can analyze image data to identifypackages in a group of objects, a stack of packages, etc. as describedabove. For example, the robotic system 100 can identify a set ofpackages, such as packages exposed to/viewable by the imaging system 160and/or accessible to the end effector 140 (e.g., packages forming thetop layer of stack 165) including the target package 112 and thesimultaneous grasp target 2250. The robotic system 100 can identify theset of packages by estimating boundaries and/or locations of theindividual packages. In some embodiments, the robotic system 100 cancompare portions of the image data to images in the master data thatrepresents known surfaces of packages. Additionally or alternatively,the robotic system 100 can perform edge-detection (via, e.g., a Sobelfilter) to detect and locate edges. The robotic system 100 can analyzethe edges to estimate boundaries of the packages depicted in the imagedata.

In some embodiments, the robotic system 100 can iteratively select oneof the identified packages as the target package 112 for subsequentprocessing/consideration. The robotic system 100 can process thepackages in the set via the iterative analysis and select a graspingcombination for transferring the packages. Alternatively, the roboticsystem 100 can transfer the target package 112 either singly or with oneor more simultaneously grasped packages (e.g., the simultaneous grasptarget 2250) at the end of each iteration.

At block 2602, the robotic system 100 can analyze a grasp set (e.g., thegrasp set 2200 of FIG. 22) for each package. The robotic system 100 cananalyze the grasp set 2200 by determining a set of available gripperpositions, such as at block 2622. Each of the grip poses in the graspset 2200 can represent a location and/or an orientation of the endeffector 140 in the real world and/or relative to the target package112. The robotic system 100 can analyze the grasp set 2200 by overlayinga model of the end effector 140 at various different grip poses over thetarget package 112 in the image data. The robotic system 100 caneliminate any of the grip poses that violate one or more predeterminedrules. For example, the robotic system 100 can eliminate the grip posesthat overlap any obstacles, such as container walls, predeterminedfixtures/structures, etc.

To reduce the processing complexity/burden, the robotic system 100 cananalyze or include a limited number of grip poses in the grasp setaccording to one or more predetermined patterns and/or rules. In someembodiments, such as illustrated at block 2624, the robotic system 100can determine a notified grasp set (e.g., a set of notified grip poses2206 of FIG. 22) for each package. Accordingly, the grasp set caninclude a set of grip poses that each align a boundary of themulti-gripper assembly (e.g., a peripheral edge of a vacuum region 117of FIG. 4) with a peripheral edge of the target package 112. Thenotified grasp set can include only the notified grip poses 2206. Therobotic system 100 can align the structures or models thereof such thatthe corresponding edges intersect or abut a vertical line or plane.

In some embodiments, the robotic system 100 can analyze the grasp set2200 based on dynamically deriving and generating the grip poses afterreceiving the image data. Alternatively, the robotic system 100 can havepredetermined instances of the grasp set 2200 for each known package.For example, the grasp set 2200 for each known or expected package canbe stored in the master data. The robotic system 100 can analyze thegrasp set 2200 based on accessing from the master data the grip posescorresponding to the identified target package.

At block 2626, the robotic system 100 can identify adjacent groupingtargets, such as additional packages (e.g., a second package, such asthe simultaneous grasp target 2250) to be considered for simultaneousgrasp with the target package 112. The robotic system 100 can identifythe grip poses that overlap with another package in the identifiedpackage set. For example, the robotic system 100 can identify the gripposes that extend along a lateral direction and over second package. Therobotic system 100 can identify the overlapped package(s) as theadjacent grouping targets. The robotic system 100 can process theidentified overlapping grip poses for simultaneously grasping theadditional package(s) along with the target package 112. In someembodiments, the robotic system 100 can prioritize the overlapping gripposes such as for evaluating the simultaneous transfers first/beforesingular transfers.

As described above, the robotic system 100 can iteratively select andanalyze the packages in the identified set of packages (e.g., the stack165 and/or a top layer thereof). The robotic system 100 can track thegrasp set for the analyzed packages, and combine the grip poses for eachof the identified packages. Accordingly, the robotic system 100 candetermine grip pose combinations for grasping and transferring theidentified packages from the start location 114 to the task location116. Each grip pose combination can represent a unique set of grip posesfor grasping the objects in the set of packages.

At block 2604, the robotic system 100 can derive combined transfercontrol settings for the planned grasps. The robotic system 100 canderive the combined transfer control settings for the overlapping gripposes. As described in detail below, the robotic system 100 candetermine the movement control parameters 2208 of FIG. 22 for the targetpackage 112, the simultaneous grasp target 2250, or any other overlappedpackages. The robotic system 100 can combine the parameters for the setof packages into one parameter corresponding to the simultaneoustransfer.

At block 2632, the robotic system 100 can identify overlapping regionsbetween the model of the end effector 140 and the packages targeted forsimultaneous grasp (e.g., the target package 112 and the simultaneousgrasp target 2250). For each of the grip poses, the robotic system 100can calculate an amount of the overlap between the end effector 140 andeach of the packages corresponding to the grip pose. In someembodiments, the robotic system 100 can categorize the overlaps such asdescribed above for FIG. 24 (e.g., ALL, SOME, and TOUCH categories)according to a set of predetermined thresholds. Alternatively oradditionally, the robotic system 100 can count the number of suctionelements 151 that overlap the packages. The robotic system 100 can countthe overall number of overlapping suction elements 151 and/or the numberfor each vacuum region 117 to represent the overlapping regions.

At block 2634, the robotic system 100 can determine control parameters(e.g., the movement control parameter 2208, such as the TSM). Thecontrol parameter can be determined for each package associated with thegrip pose. The robotic system 100 can determine the movement controlparameter 2208 based on the overlapping regions. For example, for eachof the grip poses of the end effector 140, the robotic system 100 candetermine a target control parameter that represents a force and/or arelated physical aspect of transferring the target package 112.Additionally, for each of the grip poses, the robotic system 100 candetermine a second control parameter that represents a force and/or arelated physical aspect of transferring the simultaneous grasp target2250.

According to the overlapped regions, the robotic system 100 candetermine the control parameters that represent vacuum settings forcontrolling (1) a first set of suction elements to grasp the targetpackage 112, (2) a second set of suction elements to grasp thesimultaneous grasp package 2250, and/or (3) additional sets of suctionelements to grasp further packages. In some embodiments, the roboticsystem 100 can determine the control parameters for activatingindividual instances of the vacuum regions 117 (e.g., sets of thesuction elements 151) that overlap the package. For embodiments thatallow control of individual suction elements, the control parameters canidentify the sets of the suction elements 151 that are located withinboundaries of the overlapping regions.

Additionally or alternatively, the robotic system 100 can determine thecontrol parameters that represent a measure of grip strength (e.g.,vacuum force or a number of grasping suction elements). The roboticsystem 100 can further determine the control parameters to represent oneor more aspects of transfer associated with the measure of gripstrength, such as maximum speed/acceleration and/or an estimatedtransfer time. In some embodiments, the robotic system 100 candynamically calculate (e.g., following reception of the image data) thegrip strength measure and/or the transfer aspect (e.g., the TSM) basedon the identified overlapping regions. Alternatively, the robotic system100 can calculate the control parameters offline along withcorresponding grip poses. The grasp set 2200 can include the movementcontrol parameter 2208 along with the corresponding grip pose, and therobotic system 100 can determine the parameters based on accessing thepredetermined data.

At block 2636, the robotic system 100 can generate a grouped transferparameter (e.g., a combined movement control parameter) for each grippose configured to simultaneously grasp multiple packages. For example,the robotic system 100 can generate the combined movement controlparameter based on combining into one data parameter the different setsof suction elements 151 that correspond to the overlapped packages. Insome embodiments, the combined data parameter can be a hexadecimal wordwith each bit representing an activation state of a corresponding vacuumregion/suction element. Also, for example, the combined movement controlparameter can represent a speed setting (e.g., a combined TSM) used tooperate the transfer assembly 104 and/or the end effector 140 insimultaneously manipulating/maneuvering the target package 112 and thesimultaneous grasp target 2250. In some embodiments, the robotic system100 can determine the combined movement control parameter by selecting aminimum of the transfer speeds or the TSMs for the overlapped group ofpackages (e.g., a lowest/slowest instance between the target controlparameters and the second control parameter).

At block 2638, the robotic system 100 can evaluate total transfer ratesfor transferring a set of packages. For the evaluation, the roboticsystem 100 can estimate a total transfer rate (e.g., an estimated speedor time for transferring the package set) for each of the grip posecombinations based on the corresponding control parameters. In otherwords, the robotic system 100 can determine a set of grasp solutions(e.g., grip pose combinations) for transferring the identified set ofpackages. The robotic system 100 can estimate the total transfer ratebased on the overlaps and the corresponding control parameters (e.g. theTSMs) for the grip poses in each grasp solution. The robotic system 100can estimate the total transfer rate based on combining the TSMsaccording to a predetermine process or equation. For example, therobotic system 100 can estimate the total transfer rate based on addingor averaging the TSMs.

The robotic system 100 can evaluate the total transfer rates bycomparing the total transfer rates of different grip pose combinationsor grasp solutions. Based on the comparison, the robotic system 100 canrate or rank the grip pose combinations according to the total transferrates. Accordingly, the robotic system 100 can evaluate whethersimultaneously grasping a set of packages optimizes the transfer of theoverall set of identified packages.

At block 2606, the robotic system 100 can validate the planned grasps(e.g., each grip pose). The robotic system 100 can select one of thegrip pose combinations, such as the combination with the most optimaltotal transfer rate, for validation. The robotic system 100 can validatethe grip poses within the selected combination, such as by determiningfeasibility of the grip poses according to a set of predetermined rules.

In some embodiments, the robotic system 100 can validate the plannedgrasps based on determining a release sequence for the simultaneouslygrasped packages, such as illustrated at block 2642. For a given grippose, the robotic system 100 can determine a sequence for releasing theset of simultaneously grasped packages (e.g., the target package 112 andthe simultaneous grasp target 2250) at respective target/destinationlocations. The robotic system 100 can determine the release sequenceaccording to the set of predetermined rules and/or a set ofpredetermined processes. One example of rule for determining the releasesequence can be to release the taller packages earlier than the shorterpackages. Accordingly, the robotic system 100 can minimize or eliminatedamage to the released package by minimizing a drop height for thereleased package while preventing damages (e.g., crushing events) to thelatter released packages.

As an illustrative example, the drop sequence determination process canbe based on a first approach as illustrated in block 2646. The firstapproach can be based on verifying whether the target package 112 can bereleased before the simultaneous grasp target 2250.

For the first approach, the robotic system 100 can determine a targetcontact set (P1) and a remaining set (P2) according to the overlappingregions identified for the corresponding grip pose. The target contactset can represent a set of suction elements (e.g., one or more vacuumregions 117) that overlap the target package 112 for the correspondinggrip pose. The remaining set can represent a second set of suctionelements (e.g., one or more vacuum regions 117) not in the targetcontact set. Accordingly, the first contact set and second contact setcan be mutually exclusive. The robotic system 100 can access datarepresentative of one or more physical attributes of the simultaneousgrasp target 2250 and use the accessed result to determine whether theremaining set is sufficient to grasp the simultaneous grasp target 2250.When the remaining set is verified as being sufficient to grasp thesecond package, the robotic system 100 can determine or verify that thetarget package 112 can be released before the simultaneous grasp target2250. In other words, the robotic system 100 can verify that the targetpackage 112 can be released first when the suction element(s) 151 and/orthe vacuum region(s) 117 that do not overlap the target package 112is/are sufficient to grasp the simultaneous grasp target 2250. Based onthe verification, the robotic system 100 can determine the releasesequence for releasing the target package 112 before the simultaneousgrasp target 2250.

Also, as an illustrative example, the drop sequence determinationprocess can be based on a second approach as illustrated in block 2648.The second approach can be based on verifying whether the simultaneousgrasp target 2250 can be released before the target package 112. Inother words, the robotic system 100 can change the distinction of thetarget package and the secondary/overlapped package. In someembodiments, the robotic system 100 can implement the second approachwhen the first approach fails (e.g., when the remaining set isinsufficient to grasp the simultaneous grasp target 2250).

For the second approach, the robotic system 100 can determine a targetactivation set (A1) and a second contact set (T2) according to theoverlapping regions identified for the corresponding grip pose. Thetarget activation set can represent a set of suction elements designatedto be activated to grasp the target package. The target activation setcan be less than or equal to the target contact set. The second contactset can represent a set of suction elements overlapping the simultaneousgrasp target 2250. The robotic system 100 can verify that thesimultaneous grasp target 2250 can be released before the target package112 when the target activation set and the second contact set aremutually exclusive. Accordingly, the robotic system 100 can determinethe release sequence for releasing the simultaneous grasp target 2250and the target package 112 when the second contact set and the targetactivation set are mutually exclusive.

When one or more of the simultaneous grasp poses in the grip posecombination fail to provide a release sequence (e.g., failing validationper both the first and the second approaches), the robotic system 100can select and evaluate a next grip pose combination as illustrated bythe feedback loop in FIG. 26. The robotic system 100 can validate theselected grip pose combination when the each of the grip poses thereinprovide a valid drop sequence. Accordingly, the robotic system 100 canselect the unique set/solution of grip poses (e.g., including one ormore simultaneous grasp poses for simultaneously grasping multiplepackages) for grasping and transferring the set of packages. Asdescribed above, the robotic system 100 can select the simultaneousgrasp pose that both (1) provides a valid grasp (e.g., a valid releasesequence) and (2) maximizes an efficiency measure (e.g., the transferrate and/or the transfer time) associated with transferring theidentified set of packages.

At block 2608, the robotic system 100 can derive a set of motion plansbased on the validated grasps. The robotic system 100 can derive amotion plan for each of the gripper position in the selected posecombination. For the simultaneous grasp poses in the combination, therobotic system 100 can derive the motion plans to (1) place the endeffector 140 according to the corresponding simultaneous grasp pose, (2)activate the derived/validated sets of suction elements (e.g., P1, P2,A1, and/or T2) to simultaneously grasp the overlapped packages, (3)transfer the grasped packages, and (4) release the packages at thecorresponding target locations.

In some embodiments, the robotic system 100 can derive each motion planbased on an inverse kinematics (IK) mechanism. According to the IKmechanism, the robotic system 100 can derive the motion plan based ondetermining the target location(s) for the grasped packages. The roboticsystem 100 can start from the target location(s), such as the targetlocation for the last-released package, and iterative simulate movement(e.g., by overlaying models) of the end effector 140, a robotic arm, andthe packages in a reverse travel sequence toward the start location.Accordingly, the robotic system 100 can derive a transfer path thatavoids obstacles/collisions. The robotic system 100 can derive themotion plan as the derived transfer path and/or a correspondingset/sequence of commands and/or settings for operating the end effector140 and the transfer assembly 104.

At block 2610, the robotic system 100 can implement the set of motionplans. The robotic system 100 can implement the motion plans based oncommunicating the transfer path, the corresponding commands, and/or thecorresponding settings from the processor 202 to the transfer assembly104. Upon receipt, the transfer assembly 104 can implement the motionplans according to the transfer path, execute the correspondingcommands, and/or the corresponding settings.

In some embodiments, the robotic system 100 can verify an accuracy ofthe motion plan and/or the corresponding derivations described aboveduring the implementation. For example, as illustrated at block 2652,the robotic system 100 can check one or more dimensions of thetransferred object(s) during implementation of the motion plan andbefore releasing the grasp package(s). For the check, the robotic system100 can obtain data from one or more of the destination sensors 2502 ofFIG. 25. The obtained data can represent the grasped package(s) crossinga lateral sense line/plane above the task location(s) while executingthe motion plan. Some examples of the obtained data can include triggerevents (e.g., object entry events), identifiers and/or locations of thetriggering sensors, and/or time stamps associated with the events.

Using the obtained data, the robotic system 100 can estimate whichpackage crossed the sensing line/plane and/or one or more dimensions ofthe crossing package. For example, the robotic system 100 can determinewhich of the target package 112 and the simultaneous grasp target 2250based on comparing a tracked location of the end effector 140 andrelative/tracked locations of the packages to the laterallocations/coordinates of the triggering sensors. Also, the roboticsystem 100 can calculate a height of the crossing package by subtractingthe height of the triggering sensor (a known value) from a height of theend effector 140 (e.g., a height of a bottom portion thereof) at thetime of the crossing event. Further, the robotic system 100 cancalculate a lateral dimension of the crossing package based on a numberof triggering sensors and lateral locations and/or separations betweenthe triggering sensors.

The robotic system 100 can validate the motion plan based on comparingthe dynamically derived values against expected values. For example, therobotic system 100 can verify that the crossing package is the expectedone of the target package 112 and the simultaneous grasp target 2250(e.g., the package having the greatest height amongst the simultaneouslygrasped packages). Also, the robotic system 100 compare the deriveddimension to the information stored in the master data regarding thecorresponding package.

The robotic system 100 can complete the remaining portions of the motionplan when the derived results match the expected values. When thederived results do not match the expected values, such as when anunexpected package triggers the sensors and/or the triggering packagehas unexpected dimension(s), the robotic system 100 can re-evaluate theremaining portions of the motion plan. In some embodiments, the roboticsystem 100 can derive a replacement motion plan, such as for adjustingthe release sequence/locations and/or adjusting the release height. Forexample, when the packages designated for latter release crosses thesensing line/plane before the first/earlier release package, the roboticsystem 100 can determine whether the first/early release package can bereleased at a higher height. The robotic system 100 can determine ahigher release height based on a height of the end effector 140 at thetime of the triggering event. The robotic system 100 can determine thefeasibility of the higher release height based on a height and/or aweight of the earlier release package (e.g., the target package 112).

In some embodiments, the robotic system 100 can derive the replacementmotion plan based on adjusting the release sequence/location asillustrated by the feedback loop. The robotic system 100 can derive thereplacement motion plan to release the tallest package first. Therobotic system 100 can derive the replacement motion plan without the IKmechanism. The robotic system 100 can derive the replacement motion planaccording to the data obtained from the destination sensors 2502. Therobotic system 100 can implement the replacement motion plan asillustrated in block 2610.

The method 2600 can provide and analyze a practical amount ofpermutations (e.g., the notified grasp set) for a given set of packages.Using the practical amount of permutations, the robotic system 100 canevaluate the feasibility of simultaneously grasping and transferringmultiple packages and whether that will actually improve transfer of agiven set of packages. Accordingly, the robotic system 100 canefficiently enable simultaneous grasp and transfer of multiple packagesand reduce overall transfer times for a set of packages (e.g., a stackof packages or a layer thereof).

CONCLUSION

The above Detailed Description of examples of the disclosed technologyis not intended to be exhaustive or to limit the disclosed technology tothe precise form disclosed above. While specific examples for thedisclosed technology are described above for illustrative purposes,various equivalent modifications are possible within the scope of thedisclosed technology, as those skilled in the relevant art willrecognize. For example, while processes or blocks are presented in agiven order, alternative implementations may perform routines havingsteps, or employ systems having blocks, in a different order, and someprocesses or blocks may be deleted, moved, added, subdivided, combined,and/or modified to provide alternative or sub-combinations. Each ofthese processes or blocks may be implemented in a variety of differentways. Also, while processes or blocks are at times shown as beingperformed in series, these processes or blocks may instead be performedor implemented in parallel, or may be performed at different times.Further, any specific numbers noted herein are only examples;alternative implementations may employ differing values or ranges.

These and other changes can be made to the disclosed technology in lightof the above Detailed Description. While the Detailed Descriptiondescribes certain examples of the disclosed technology as well as thebest mode contemplated, the disclosed technology can be practiced inmany ways, no matter how detailed the above description appears in text.Details of the system may vary considerably in its specificimplementation, while still being encompassed by the technologydisclosed herein. As noted above, particular terminology used whendescribing certain features or aspects of the disclosed technologyshould not be taken to imply that the terminology is being redefinedherein to be restricted to any specific characteristics, features, oraspects of the disclosed technology with which that terminology isassociated. Accordingly, the invention is not limited, except as by theappended claims. In general, the terms used in the following claimsshould not be construed to limit the disclosed technology to thespecific examples disclosed in the specification, unless the aboveDetailed Description section explicitly defines such terms.

Although certain aspects of the invention are presented below in certainclaim forms, the applicant contemplates the various aspects of theinvention in any number of claim forms. Accordingly, the applicantreserves the right to pursue additional claims after filing thisapplication to pursue such additional claim forms, in either thisapplication or in a continuing application.

What is claimed is:
 1. A method for operating a transport robot, themethod comprising: receiving image data depicting packages at a startlocation, wherein the image data is for analyzing simultaneous grasp oftwo or more of the packages; identifying a target package based on theimage data; determining a grasp set representative of grip posesrelative to the target package; identifying a set of grip poses withinthe grasp set that overlap a second package; deriving a combinedmovement control parameter for simultaneously manipulating the targetpackage and the second package, wherein deriving the combined movementcontrol parameter includes: determining a target control parameter basedon the grasp set, wherein the target control parameter represents aforce, an acceleration, a rate, or a combination thereof associated witha speed of transferring the target package according to a target pose ofthe multi-gripper assembly; determining a second control parameter basedon estimating an overlap between the multi-gripper assembly and thesecond package, wherein the second control parameter represents a force,an acceleration, a rate, or a combination thereof associated with aspeed of transferring the second package according to the target pose ofthe multi-gripper assembly; and selecting the combined movement controlparameter as the target control parameter or the second controlparameter corresponding to a lower transfer speed; determining a releasesequence for placing the target package and the second package atrespective target locations; selecting a simultaneous grasp pose fromthe grasp set based on the combined movement control parameter and therelease sequence, wherein the simultaneous grasp pose represents a poseof a multi-gripper assembly of the transport robot for simultaneouslygrasping and transferring the target package and the second package;deriving a motion plan based on the simultaneous grasp pose, the motionplan for simultaneously maneuvering the multi-gripper assembly totransfer the target package and the second package to the respectivetarget locations; and implementing the motion plan for operating thetransport robot to simultaneously grasp and transfer the target packageand the second package.
 2. The method of claim 1, wherein the grasp setis a set of notified poses, wherein each notified pose is for aligning aboundary of the multi-gripper assembly with a peripheral edge of thetarget package.
 3. The method of claim 1, wherein the combined movementcontrol parameter represents a speed setting for the transport robotand/or the multi-gripper assembly for maneuvering the target package andthe second package simultaneously.
 4. The method of claim 1, wherein thecombined movement control parameter represents vacuum settings forcontrolling a first set of suction elements and a second set of suctionelements in the multi-gripper assembly to grasp the target package andthe second package, respectively.
 5. The method of claim 1, whereinselecting the simultaneous grasp pose includes selecting thesimultaneous grasp pose that maximizes an efficiency measure associatedwith transferring a set of packages that includes the target package andthe second package.
 6. The method of claim 5, further comprising:identifying the set of packages based on the image data, wherein the setof packages represents objects that are both exposed to an imagingdevice associated with the image data and that are targeted for transferbetween corresponding start and task locations; based at least partiallyon the grasp set, determining pose combinations that each represent aunique set of grip poses for grasping and transferring the objects inthe set of packages; estimating a total transfer rate for each of thepose combinations based on associated movement control parameters; andwherein: the simultaneous grasp pose is selected based on selecting oneof the pose combinations having a lowest instance of the total transferrate, wherein the total transfer rate represents a duration fortransferring the objects and/or the set of packages.
 7. The method ofclaim 5, wherein the set of packages represents packages that form a toplayer of a package stack.
 8. The method of claim 1, wherein determiningthe release sequence includes: determining a target contact setrepresenting a set of suction elements overlapping the target package;determining a remaining set representing a second set of suctionelements outside of the target contact set; verifying whether theremaining set is sufficient to grasp the second package based on one ormore physical aspects of the second package; and determining the releasesequence for releasing the target package before the second package whenthe remaining set is verified as being sufficient to grasp the secondpackage.
 9. The method of claim 1, wherein determining the releasesequence includes: determining a target activation set representing aset of suction elements designated to grasp the target package;determining a second contact set representing a second set of suctionelements overlapping the second package; verifying whether the secondcontact set and the target activation set are mutually exclusive; anddetermining the release sequence for releasing the second package beforethe first package when the second contact set and the target activationset are mutually exclusive.
 10. The method of claim 1, wherein therelease sequence is determined based on a rule for releasing first atallest of grasped packages.
 11. The method of claim 1, wherein derivingthe motion plan includes: deriving placement locations for the targetpackage and the second package according to the release sequence; andderiving the motion plan based on iteratively simulating and verifyingmovement of the target package and the second package in a reversesequence from the placement locations to starting locations of thetarget package and the second package.
 12. The method of claim 1,further comprising: obtaining data from one or more destination sensors,wherein the data represents the target package and/or the second packagecrossing a lateral line above a task location during execution of themotion plan; identifying a set of locations based on the obtained data,wherein the set of locations locate the one or more destination sensors;and determining a validity of the motion plan based on the set oflocations.
 13. The method of claim 12, wherein validating the motionplan based on the set of locations includes: identifying a crossingpackage based on the set of locations; deriving at least one estimateddimension of the crossing package, wherein the at least one estimatedimension includes an estimated height and/or an estimated lateraldimension of the crossing package; and validating the motion plan (1)when the crossing package matches either the target package or thesecond package initially estimated as having a longer height and/or (2)when the at least one estimated dimension matches one or morecorresponding dimensions of the target package or the second package asstored in master data.
 14. The method of claim 12, further comprising:deriving a replacement plan when the motion plan is determined to beinvalid according to the set of locations, wherein the replacement planis for manipulating the multi-gripper assembly to place the targetpackage and the second package at respective task locations, and thereplacement plan is derived according to the data obtained from the oneor more destination sensors for replacing a remaining portion of themotion plan; and in response to the invalid determination, implementingthe replacement plan instead of the remaining portion of the motion planfor placing the target package and the second package at the respectivetask locations.
 15. A system configured to control operation of atransport robot, the system comprising: a communication circuitconfigured to communicate data, commands, and/or settings with a set ofsensors and the transport robot, wherein the communication circuit isconfigured to: receive image data depicting packages, including a targetpackage and a second package, at a start location, and send a motionplan or a set of corresponding commands and/or settings for operatingthe transport robot to simultaneously grasp and transfer the targetpackage and the second package; at least one processor coupled to thecommunication circuit and configured to: identify the target packagebased on the image data; determine a grasp set representative of gripposes relative to the target package; identify a set of grip poseswithin the grasp set that overlap the second package; derive a combinedmovement control parameter for simultaneously manipulating the targetpackage and the second package, wherein deriving the combined movementcontrol parameter includes: determining a target control parameter basedon the grasp set, wherein the target control parameter represents aforce, an acceleration, a rate, or a combination thereof associated witha speed of transferring the target package according to a target pose ofthe multi-gripper assembly; determining a second control parameter basedon estimating an overlap between the multi-gripper assembly and thesecond package, wherein the second control parameter represents a force,an acceleration, a rate, or a combination thereof associated with aspeed of transferring the second package according to the target pose ofthe multi-gripper assembly; and selecting the combined movement controlparameter as the target control parameter or the second controlparameter corresponding to a lower transfer speed; determine a releasesequence for placing the target package and the second package atrespective target locations; select a simultaneous grasp pose from thegrasp set based on the combined movement control parameter and therelease sequence, wherein the simultaneous grasp pose represents a poseof a multi-gripper assembly for simultaneously grasping and transferringthe target package and the second package; and derive the motion planbased on the simultaneous grasp pose, the motion plan for maneuveringthe multi-gripper assembly to simultaneously transfer the target packageand the second package to the respective target locations.
 16. Thesystem of claim 15, wherein the processor is configured to derive themotion plan for operating a first set of suction elements of themulti-gripper assembly and a second set of suction elements of themulti-gripper assembly, wherein the first set is configured to grasp thetarget package and the second set is configured to grasp the secondpackage.
 17. A tangible, non-transitory computer-readable medium havingprocessor instructions stored thereon that, when executed by one or moreprocessors, cause the one or more processors to perform a method, themethod comprising: receiving image data depicting packages at a startlocation, wherein the image data is for analyzing simultaneous grasp oftwo or more of the packages; identifying a target package based on theimage data; determining a grasp set representative of grip posesrelative to the target package; identifying a set of grip poses withinthe grasp set that overlap a second package; deriving a combinedmovement control parameter for simultaneously manipulating the targetpackage and the second package, wherein deriving the combined movementcontrol parameter includes: determining a target control parameter basedon the grasp set, wherein the target control parameter represents aforce, an acceleration, a rate, or a combination thereof associated witha speed of transferring the target package according to a target pose ofthe multi-gripper assembly; determining a second control parameter basedon estimating an overlap between the multi-gripper assembly and thesecond package, wherein the second control parameter represents a force,an acceleration, a rate, or a combination thereof associated with aspeed of transferring the second package according to the target pose ofthe multi-gripper assembly; and selecting the combined movement controlparameter as the target control parameter or the second controlparameter corresponding to a lower transfer speed; determining a releasesequence for placing the target package and the second package atrespective target locations; selecting a simultaneous grasp pose fromthe grasp set based on the combined movement control parameter and therelease sequence, wherein the simultaneous grasp pose represents a poseof a multi-gripper assembly for simultaneously grasping and transferringthe target package and the second package; deriving a motion plan basedon the simultaneous grasp pose, the motion plan for simultaneouslymaneuvering the multi-gripper assembly to transfer the target packageand the second package to the respective target locations; andimplementing the motion plan for operating a transport robot tosimultaneously grasp and transfer the target package and the secondpackage.
 18. The tangible, non-transitory computer-readable medium ofclaim 17, wherein the method further comprises: identifying a set ofpackages based on the image data, wherein the set of packages includesthe target package and the second package, wherein the set of packagesrepresents objects that are both exposed to an imaging device associatedwith the image data and that are targeted for transfer betweencorresponding start and task locations; based at least partially on thegrasp set, determining pose combinations that each represent a uniqueset of grip poses for grasping and transferring the objects in the setof packages; estimating a total transfer rate for each of the posecombinations based on associated movement control parameters; andwherein: the simultaneous grasp pose is selected based on selecting oneof the pose combinations having a lowest instance of the total transferrate, wherein the total transfer rate represents a duration fortransferring the objects and/or the set of packages.
 19. The tangible,non-transitory computer-readable medium of claim 17, wherein determiningthe release sequence includes: determining a target contact setrepresenting a set of suction elements overlapping the target package;determining a remaining set representing a second set of suctionelements not in the target contact set; verifying whether the remainingset is sufficient to grasp the second package based on one or morephysical aspects of the second package; when the remaining set isverified as being insufficient to grasp the second package, determininga target activation set representing a set of suction elementsdesignated to grasp the target package, wherein the target activationset includes less elements than the target contact set; determining asecond contact set to represent a set of suction elements overlappingthe second package; verifying whether the second contact set and thetarget activation set are mutually exclusive; and determining therelease sequence for releasing the second package before the firstpackage when the second contact set and the target activation set aremutually exclusive.